Why when training ML models do we not take the model with the highest accuracy

Question

Its pretty common for an ML model to lose accuracy at various points during training. But presumably that means they are worse so why do we take the last version instead of the one that had the highest accuracy?

In: 1

Answer 1

I think the measurement of accuracy is not accurate. You are measuring something like the true accuracy + noise. The accuracy of model is really just showing how well it does with whatever validation set you use, which is subset of all tasks you are going to give it. If all you ever wanted to do was use the model against the validation set, then sure, select the model that shows highest accuracy with the validation set, I guess.