It's sad but sometimes you have to at lest project the image that you are willing to dive to the same level as your attackers, I've found this is usually enough, if not then you have to be willing to Run, fight or simply stay away from the area where this type of thing happens, i feel for you, not feeling safe in your own home town sucks for sure.
There are places where I live I would not go willingly but fortunately those areas are few and far between. The good thing is that most of these people are all bluff and no action but it's difficult to know which is which.