If that was the only thing he did, imo, he'd catch less.
He also was either racist or "culturally insensitive" to Dominicans and other people faking being black (his words).
He also thinks it is cool to use his position to impose his beliefs on others, but NOT cool to be asked about that, and is very rude to anyone that does. Can't have it both ways, if you put it out there, you have to be willing to talk about it, not call people pricks (his words) over and over in public.
I have no idea what is "too much" or "unfair" in how people feel about him. For me, if it was just a guy being wrong about SSM, that would be very different than actively campaigning for it, then being a total jerk when people ask him about it. Oh, and the whole "racist" thing.