Some materials are better at transporting heat than others.
A material with a higher thermal conductivity will suck the heat from you more quickly and will thus feel colder.
[edit] To be a little bit clearer: You are only able to feel something is hot or cold because your are echanging heat with it, which is lowering or raising the temperature of the sensors in your skin. If the object as a low conductivity and “refuses” to transfer heat from/to you, it will feel closer to ambient because your sensors won’t change temperature.
Latest Answers