Scaling machine learning up
What changes when the dataset no longer fits in memory or training takes hours: algorithm costs, mini-batches, online learning, sparse matrices, spreading the computation and batch prediction.
- 23 cards
- 5 topics
- Free
The outline, topic by topic
Each topic is weighted the way the exam weighs it.
- Concepts
- Questions
- Sort and link
- The numbers
- Scenarios
A few cards from the deck
Time complexity: what is it?
Show the answer
How an algorithm's computing time grows with the size of the data: proportional to the number of rows, to its square, and so on.
It predicts what will happen with 10 times more data: a linear cost is multiplied by 10, a quadratic cost by 100.
What does mini-batch learning bring when the training set does not fit in memory?
- It reads the set in small packets and keeps only one in memory at a time
- It compresses the whole training set in memory without losing any value
- It removes the least useful rows so that the remaining set fits in memory
- It trains several independent models on the whole set, then keeps the best one
Show the answer
It reads the set in small packets and keeps only one in memory at a time
At each step, the gradient is computed on one mini-batch only: memory depends on the batch size, not on that of the set. No row is thrown away: all are read, each in turn.
Computing the distance between all pairs of points takes a time that grows as the … of the number of points.
Show the answer
Computing the distance between all pairs of points takes a time that grows as the square of the number of points.
n points form about n²/2 pairs: 10 times more points, about 100 times more pairs. This is what quickly makes hierarchical clustering or a full distance matrix impractical.
Mini-batch gradient descent: what is it?
Show the answer
Training that corrects the model's parameters after each small packet of rows, instead of waiting to have read the whole set.
Only one mini-batch must fit in memory at a time, and the parameters progress from the first rows. It is the base method of neural networks.