Because higher magnitude quakes (6 or greater) usually occur, on average, about every 20 - 30 years. However, there are literally thousands of lower magnitude quakes that occur throughout the state every year that are not felt by humans. Last year and the year before there were at least two I remember that were great enough to be felt in my area, though no property damage was reported.
In the U.S. California is second only to Alaska in the number of earthquakes the state experiences annually. More importantly, besides a quake's magnitude and ability to cause damage, one must also consider location, depth and proximity to populated regions. Obviously, with California being the most populated state, even lower magnitude quakes can cause more damage than a higher magnitude quake in a less densely populated region such as Alaska.
To assume non-damaging earthquakes 'do not really matter' is a bit naive to say the least. The damage we see above ground, is of course, second to what is occurring under the Earth's surface. To fully understand the damage/changes which can be a result of any seismic activity, I suggest you do some reading on plate tectonics, types of fault lines/fractures, changes in the Earth's heat/pressure, elasticity, etc.