The former is always fine, the latter is usually not. Occasionally people are understandably a bit confused. Not everyone realizes that they are "up" enough that they are capable of punching down. You might not be a member of the Yankees, but even just being a Yankees fan means you're probably punching down.
