The mathematics of learning
The tools that make a model learn: dot product, loss function, gradient, learning rate, gradient descent, chain rule, backpropagation and softmax.
- 23 cards
- 3 topics
- Free
The outline, topic by topic
Each topic is weighted the way the exam weighs it.
- Concepts
- Questions
- The numbers
A few cards from the deck
Loss function: what is it?
Show the answer
A single measure of the gap between a model's predictions and the true answers, which training tries to make as small as possible.
It turns “learning well” into a number that can be differentiated. Mean squared error is chosen for a value to predict, cross-entropy for a class.
Which value of w do we get after two steps of this gradient descent?
```python
# f(w) = (w - 3) ** 2, donc f'(w) = 2 * (w - 3)
w = 0
pas = 0.25
for _ in range(2):
w = w - pas * 2 * (w - 3)
print(w)`2.251.53.0-2.25
Show the answer
2.25First step: gradient −6, w = 0 + 0.25 × 6 = 1.5. Second step: gradient −3, w = 1.5 + 0.75 = 2.25. After one step only, it would be 1.5; the minimum, 3, is approached but not reached.
What a neuron computes before its activation is a … of the weights and the inputs, to which the … is added.
Show the answer
What a neuron computes before its activation is a dot product of the weights and the inputs, to which the bias is added.
With weights (2, −1) and inputs (3, 4): 2 × 3 + (−1) × 4 = 2, before the bias.
Gradient: what is it?
Show the answer
The vector of the loss's derivatives with respect to each parameter; it points in the direction in which the loss grows fastest.
To lower the loss, you move in the direction opposite to the gradient.
We minimise f(w) = (w − 3)² from w = 0 with a step of 1.1. What does w do over the first three steps?
- It approaches 3, slightly overshooting the target, then settles
- It jumps from one side of 3 to the other, moving a little further away at each step
- It reaches 3 in a single step, then stays still in the following steps
- It moves ever more slowly towards 3, never overshooting it
Show the answer
It jumps from one side of 3 to the other, moving a little further away at each step
Each step multiplies the gap w − 3 by 1 − 2 × 1.1 = −1.2: a factor with absolute value above 1, so the gap grows while changing sign (w is 6.6, then −1.32, then 8.18). A step of 0.5 would reach 3 at once, a step of 0.1 would approach it gently.