I'd argue the exact opposite of what you said.
I don't want the language to be needlessly offensive, but what's really important here is that a single mistake doesn't label you.
Take, for example, a girl who just got dumped by her boyfriend and says "All men are just assholes." (I've heard it. Many, many, many, many times.)
The language is needlessly offensive. She could have said his name instead. But no, she says "men". That's sexist, is it not?
Or perhaps we use another phrase, more related to this thread.
A woman, after her husband forgets to put the big spoons separately from the small spoons, says exasperatedly, "Stupid man.", takes the spoon, and does it for him.
I mean, according to you(I think it was you who said that), there is no reason to include gender in that phrase unless it's used in a sexist way to demean that gender.
Should we label that woman as sexist, in the name of reducing needlessly offensive language?
I could go on. But I think my point is clear. I really believe that allowing a mistake to label someone is far worse than there being needlessly offensive language.
And if it's sexist, as it obviously is, and the speaker made a mistake when saying that, should we label him as sexist? (This is assuming you agree with Phi's statement. If not, ignore this.)