Thanks in advance, I wants to understand the concept and not just know the equations!
In: 8
I know you’re asking about the physics concepts, but you might also be interested to know that information theory exists, and has its own concept of entropy!
In that world, entropy is a measure of the evenness of the distribution of probability of encountering symbols – and correspondingly, the disorganization of information.
If you roll a fair die a bunch of times, you have to write down the entire sequence to identify it uniquely, and there’s not really a great way to improve on that. If your die is loaded? There are schemes that do better on average, and only take more work when something unusual happens. Imagine if you had a loaded die that favored 1, and so you rolled 20 1s, a 4, and then 10 more 1s- I just described that long sequence in a really compact way, right?
If your mind goes to compression, you’re right on the money. This is the principle that lossless compression schemes like Zip (LZW) use – while there are lots of schemes to shrink data that’s organized in some way, you don’t benefit from those schemes if the entropy is high, and so some data are going to compress better than others!
Latest Answers