Training, evaluating, overfitting

The cycle of a training run: training, validation and test sets, epoch and batch, error curves, underfitting, overfitting and their remedies.

The outline, topic by topic

Each topic is weighted the way the exam weighs it.

Overfitting: what is it?
Show the answer
The flaw of a model that memorises the detail and noise of its training examples, and so errs on new data.
Typical sign: the training error keeps falling while the error on unseen data rises again.
A team tries fifty settings and keeps the one with the best score on the test set, which it reports as the model's performance. What is the problem?
- The test set was used to choose: the reported score is too optimistic
- The test set was too big: a large set makes the measure unstable from one trial to the next
- Fifty trials are not enough to seriously compare settings with each other
- A test score can never exceed the training score of the same model, so the reported figure is impossible
Show the answer
The test set was used to choose: the reported score is too optimistic
Choosing the best on the test set makes it play the role of validation: the chosen setting has adapted to its quirks. The choice belonged on validation, with a single measure on test afterwards.
The three sets each have a role: … adjusts the weights, … tunes the settings, … measures final performance, once only.
Show the answer
The three sets each have a role: training adjusts the weights, validation tunes the settings, test measures final performance, once only.
Measuring on data already seen, or consulting the test set repeatedly, gives an over-optimistic performance.