I never advocate violence. I hate armchair revolutionaries, the people who call for the revolution but never quite manage to lead it themselves. But the idea that violence is never thought to be wrong in our society is absurd. We lionize our combat veterans. We praise our cops even when they engage in unjustified violent behavior. The white vigilante in many forms is a hero in pop culture (shooting black criminals, fighting against imagined government tyranny).
There's nothing we exalt more than WWII veterans. They were sent off to hell to kill Nazis. Tom Brokaw's meal ticket is praising the Greatest Generation. Beating Nazis is the 20th century's foundation myth of the US.
More importantly, Nazis aren't just people who are a bit racist, like your typical conservative. The are exterminationists. They explicitly ally themselves with those who deliberately committed genocide on an unimaginable scale (colonial and imperial atrocities are not entirely dissimilar, but the technocratic genocidal program of Hitler was somewhat unique). Someone who wears a Swastika isn't simply saying "I don't much like Jews and Blacks." They are saying they are on board with a program to exterminate them.
Should you punch them? I dunno. Captain America would. That's why he's a comic book hero.