I haven't seen Girls and have no opinion about anything to do with the show, and don't quite understand why that show in particular has inspired the latest round of discussion about diversity on the teevee, but I got hit with a cluestick years ago when someone pointed out that the issue wasn't simply that all of the lead actors of some shows were white. That might be problematic, but the kind of problem which can be justified or explained away (not necessarily satisfactorily explained, but that there were explanations). A major issue was that you had these shows that would take place in San Francisco, Los Angeles, or New York (for example), and all of the background characters - the extras - were white.
"All" is generally an exaggeration, but the point is that even when shows are set in places with significant racial diversity, it's often the case that the world they portray, and not just in the foreground, is a lot whiter than reality.