The walls of your house are touching the outside air and will be at a temperature between your inside air (65) and the outside (hot or cold). Everything that has a temperature will emit some heat as radiation, mostly infrared, and how much is determined by the temperature of the object. You and your walls are always exchanging heat this way and when they are colder you feel less of this radiation reaching you making you colder. I did the math at one point but I don’t remember the exact answer anymore. It’s noticable.
Latest Answers