Evaluating a model
Saying what a model is worth on data it has not seen: train-test split, cross-validation, confusion matrix, accuracy, precision, recall, F1, ROC curve, and for regression MAE, RMSE and R².
- 30 cards
- 6 topics
- Free
The outline, topic by topic
Each topic is weighted the way the exam weighs it.
- Concepts
- Questions
- Sort and link
- The numbers
- The curves
- Scenarios
A few cards from the deck
Train-test split: what is it?
Show the answer
The splitting of the set into two parts: one to learn from, the other kept aside to measure the model on data never seen.
A model scored on its own training data always looks good, even if it learned everything by heart. The test measures the ability to generalise, and is used once, at the end.
In a 5-fold cross-validation, how many models are trained and on what share of the set does each train?
- Five models, each on 80% of the set
- A single model, trained on 80% of the set then tested five times on the remaining 20%
- Five models, each on 20% of the set, the single fold assigned to it in that round
- Twenty-five models: each fold is used for training with each of the four other folds
Show the answer
Five models, each on 80% of the set
In each round one fold (20%) is the test and the other four (80%) the training, with a new model each time. Five rounds, five scores, then their average.
… is the share of correct predictions among all predictions: (TP + TN) / total.
Show the answer
Accuracy is the share of correct predictions among all predictions: (TP + TN) / total.
It is misleading when one class dominates: with 1% positives, always answering “negative” gives 99% accuracy while detecting nothing.