This is a bit complicated, mostly because a pigment can be black as far as visible light is concerned, but completely transparent to infra-red light.
Secondly, colour only affects radiated heat but when it comes to clothing conduction and convection are also important. There is unlikely to be much radiative transfer between your skin and clothing – this will be dominated by conduction. In effect it is only the colour of the very outer layer that matters.
Also you need to consider which way the heat is going; generally from your body to the atmosphere. As such you want your clothes to minimise heat transfer if you want to stay warm. In that respect white clothing would be preferable to black (with the caveats above).
Latest Answers