Its pretty common for an ML model to lose accuracy at various points during training. But presumably that means they are worse so why do we take the last version instead of the one that had the highest accuracy?
In: 1
Another comment explains overfitting which is one reason. Another reason is that you might be trying to predict something rare.
Let’s say you’re making pencils and you want the computer to tell if a pencil is defective. Let’s say 1 out of 1000 pencils is defective. Your highest accuracy model might be the one that always guesses the pencil is good, because that will be right 99.9% of the time. But that model isn’t really doing anything!
A similar thing happens when trying to identify cancer cells or other health problems. You really don’t want to miss the rare cases when there’s a problem.
There are other numbers to look at, like true positive rate, true negative rate, positive predictive value, and negative predictive value. You can think of these kinda like accuracy for just one case, yes accuracy and no accuracy. Which one you should use depends on what you’re doing.
Latest Answers