It’s a different answer for each type of fiber and each method of making fabric.
Wool fabric shrinks in agitated hot water (like your washing machine) because the fibers in the yarn it is woven or knit from come un-spun a bit and recombine in a different way to lock together. This is called fulling and is an important part of producing wool clothes. Sometimes the finished fabric is called boiled wool. It can’t stretch back out again.
Cotton weave, like non-stretchy jeans, unspins in a similar way to wool when washed, but because it is made of cellulose instead of proteins it doesn’t lock in a different way. It can return to its original size, to an extent (also called “ease”). Cotton knit (like tshirts) is already stretchy, and when you stretch it in one direction it gets smaller in the other direction. Washing rearranges the knit structure back to the way it originally was.
This is way oversimplified, it’s a really complicated question involving the microscopic features of different types of fabric.
Latest Answers