Yes, I worry about it. However, as Jacob said, I also worry about safety when driving. I know personally far more people who have been killed/severely injured in car accidents than bike accidents. I have had one "bad" biking injury where I had to get out of the way of a car quickly and wound up slamming against a wooden railing, which scraped several layers of skin off my shin. Ouch, but nothing that could not be cleaned up at home.
I do live in an area with a lot of cyclists so cars are, I think, more cyclist-aware.
In terms of safety as a woman, I feel far safer on a bike than walking--you are moving much faster than anyone on foot is. I have had some nasty incidences with guys following me in cars when I have been on foot. This has never happened on my bike. That said, I do take some precautions. There are some areas where I will not bike alone at night. If I'm going anywhere further than the grocery store I do figure out what my options are for getting home before leaving. If I don't like any of them, I don't go. This has yet to severely limit my social life (I do live near a bus line, so this can be a good alternative to biking at night). I also have several neighbors with cars who could be called on to pick me up if I found myself unexpectedly stranded somewhere. I am always insistent on giving people $$ for gas if they give me a ride so I don't feel like this is really freeloading, plus I am happy to help them out in other ways if they need it. If I were facing a really unpalatable ride home for some reason and couldn't get public transport or a friend to pick me up, there is always the last-ditch option of a cab, which taken on a very occasional basis is still cheaper than owning a car.
Also, I know it doesn't necessarily help when you are biking down a dark street at 2AM, but in most sexual assaults the victim knows their attacker. You are far more likely to be raped by someone you know than by a random stranger on the street.
