I'm putting my political science hat on for a minute. Why does whitey fear people of color? Because historically, the more "white" your community was, the more affluent it was. Now there's certainly a bit of racism mixed in there, but some might argue it's a response to a relatively functional psychological defense mechanism. There is a direct statistical correlation between %Caucasian and a community's median home value. Majority-minority neighborhoods have a strong statistical tendency to be in lower socioeconomic areas. So when a white individual sees minority encroachment in their neighborhood, they associate it with a respective decrease in their property values and overall quality of life. If minority encroachment is feared, it's perhaps not for something inherently wrong about minority individuals themselves, but rather the fear that the white person's quality of life was at risk.
The best example of this is the cyclical nature of urban sprawl over the last 60 or so years. As minority ranks grew in the inner cities, Whites "sprawled" into the suburbs and pushed further out. The people who left were also those with the greatest financial resources (allowing them to pick up and leave). Those remaining were, by their inability to leave, inherently less well off. But eventually lower SES groups pushed outward as well, driving the affluent further and further out. Eventually, in the 2000s, we began to see sprawl turn inward, with affluent citizens moving back into the central cities that had largely been abandoned. Many would speculate that this is a cyclical pattern that will continue for some time (in my theory, until most work becomes entirely remote, and people can live wherever they want).
By way of self-fulfilling prophecy, white communities have often purposely resisted apparent minority encroachment as a way to maintain their QOL and property values. No reasonable person would think that a Black or Hispanic family moving in next door will have a direct, negative impact on the community. But there is a statistical precedence for feeling that a community "becoming" Hispanic or Black is a sign of negative growth. So, is it wrong for a White American to resist racial integration in their community? I would say so. Is it irrational? That's a tougher question. I mean how do you argue with a phenomena that is driven by statistical data that is accurate, if not misunderstood.
Forgive me if I myself come off a bit racist - I haven't eaten, but I did just drink a Duvel.