So having read this a few times over now and taken a day or two to mull it over, (feel free to correct me if I'm wrong) a simplified version of what you said is this:
1)Some things are easier to oppose because society has largely agreed on these things over time.
2)Free speech is a privilege and there's nothing new about it being weaponized, I'm just a bit inexperienced in it. (no argument from me, just summarizing)
3)The government abstaining from regulating speech comes with costs, mostly to the disenfranchised. Here are a few that are very hard to argue with.
Point by point:
1) I don't think the flags are easier to oppose because of what they symbolize (their symbolism is exceedingly easy to oppose) I think they're easier to oppose because they are flags. They're objects in meatspace that I can tell at a glance if they are or aren't nazi flags. I could write a document with picture of a swastika, put it in a law book and have it handed down and boom now they're illegal to display publicly.
It wouldn't be a perfect fix because what about all the minor modifications you could make to one before it stops being a hate symbol? Is it enough to change the colors or the angles? what about a plus sign with little accent marks? etc.
It's certainly easier to outlaw than the nebulous concept of "being a nazi" because that's a crime that exists in your head. If I could I would but it's not possible.
2) The idea idea that free speech isn't universal (within places where it is law) has never occurred to me until you put it forth here. I still need to consider the implications but for the moment I'm going with my gut reaction which is that doesn't mean that it's not a right, even if some are unable to make use of it due to lack of privilege.
3) So this is a value judgement. I don't think it's objective, only subjective. We as a society have to decide if we want to live in the world we live in now or the world where the government attempts to fix these problems. Given the volatility of our democracy even if I did trust the people in charge, I wouldn't want them to make these changes because the next people may not be so trustworthy.
It's a bit of a lose lose in my view. Do we lose more now with the issues you outlined above or would we lose more if we tried to fix them and had to live with what that ended up looking like? I put my money on the latter.