As is occasionally pointed out when journalists and news business people complain that Google is stealing their content, if they don't want Google to index their pages they can simply... tell Google not to index their pages by inserting a bit of
code into them. What they really want Google to do is pay them for the privilege of making money from a derivative of their product, the way book reviewers always pay novelists, for example.
