In any case the distinction is between “passing off the work of others as your own” and “making references that you expect your audience to be educated enough to understand.”
Yes of course pure plagiarism is bad, but I also think that making unattributed references in, say, a political speech is much more problematic than it used to be. Perhaps because digital culture is making people think about issues they haven't before, I think we're actually getting a bit stupider about such things. If I were a politician I wouldn't make anything but the most obvious references in a political speech for fear of being accused of plagiarism, while back in the day making unattributed allusions and references was what good literature and speechwriting entailed.