In the suburb/exurb of my teen years, there was one black family living in the neighborhood. And, yes, everybody in the neighborhood knew precisely which house they lived in. I'm not saying they faced
any overt hostility (maybe they did? don't know), or were excluded from the social activities of the neighborhood, though it wasn't a neighborhood with many neighborly social activities. But there was a black family. We knew where they were.