What is Overfitting in machine learning and why is it bad?
In: 2
Imagine you do an experiment. You find somebody from every age from 1 to 100 and you measure their height. Then, you plot these on a graph. Experience tells you that for the first 20 years or so (probably less, but let’s roll with it) you get taller and taller. This happens quickly at first, but slows down as you approach 20. Then, your height stays flat for the next 50 years or so, until 70. Then, beyond that, you begin to lose a bit of height. The “line of best fit” of the data you’ve collected should fit that pattern. It should be a smooth curve that peaks around 20 and plateaus for a while, before gradually dropping at the end.
Say instead you tread your data like a dot-to-dot. You connect 1 to 2 to 3 to 4… with straight lines and sharp corners. This would be overfitting the line. Instead of seeing the whole smooth progression, your line might make you think or heights go up and down constantly. Maybe you happened to pick a tall, early-developing 12 year old and then a short, late-developing 13 year old. The line you’ve drawn makes it look like we peak at 12, then immediately shrink, before slowly growing again. Perhaps the 50s were all shorter women and the 60s taller men, or maybe they even alternated!
Point is, you’re giving too much weight to each individual datapoint rather than to the general trend.
This is similar to overfitting in machine learning. Every part of your training dataset has certain flukes and random features. If you train on too small a set for too much, you end up with a system that is very good at dealing with the training images but not so good at stuff beyond that.
Latest Answers