It’s a misconception that “infrared light = heat”. *Any* object with heat will radiate some kind of light depending on its temperature. It just happens to be the case that at human temperatures, the light is infrared.
The sun is very hot, and as a result gives off a lot of visible light. If everything were surface-of-the-sun temperatured, then we’d think of visible light as being equal to heat.
The reason black shirts warm you up is because they absorb the visible light from the sun. Absorbing energy of any kind will heat you up, so the black shirt heats up.
If I had to guess, there would be no noticeable difference between wearing a black shirt vs a white shirt under your coat.
Latest Answers