Am I wrong for defending myself as a man?

You are not wrong at all for that. It's just another one of those stupid double standards with genders, and frankly, I say good for you if you're willing to fight that.