Because of skin color can be indicative of where one is from, and humans often value racial loyalty.
Ego maybe, they are X color, so they value that like themselves,
Knowledge maybe, their family and immediate genetics are X color, loyalty to family,
There are many labels, but genetics is one, and it often expounds outward.
Self, family, tribe, humans, animals, insects, rocks.
Sickness response maybe, that which breaks a norm is medical danger,
Skin often changes color with various diseases, many of them spreadable.
Tribalism again, we are better able to read and understand the faces we grew up with.
Shape is not color exactly, but color influences view, and shape and color often go together.
. . .
I imagine before modern technology, skin color was much more useful.
Light skin better at getting what we need from the sun in cloudy climate,
Dark skin better at blocking harmful sun in sunny climate.
White skin better for hiding in the snow, though, I'm not sure that one ever applied to humans.
One see's many arctic animals with white fur, I do not recall any white skin animals in the arctic,
Well maybe some marine life, unsure.
. . .
Tribalism again,
Some parts of India practice skin whitening I have heard,
People value their culture, heritage, self, history, wrap it up in their ego.
Easy for skin whitening to be seen as rejection of self, of people.
Course, many people in America tan their skin,
Some in Japan gyaru look.