Gradient descent
Minimising a function by following the slope: derivative and gradient, update rule, choice of step, convergence, oscillation and divergence, local minimum and stopping criterion.
- 24 cards
- 5 topics
- Free
The outline, topic by topic
Each topic is weighted the way the exam weighs it.
- Concepts
- Questions
- Sort and link
- The numbers
- The curves
A few cards from the deck
Gradient descent: what is it?
Show the answer
A method that looks for a minimum by starting from a point and moving at each step in the direction where the function decreases fastest.
The rule is x ← x − step × f′(x) for one variable. It needs only the local slope, not the whole shape of the function.
f(x) = x² − 4x, step 0.25, start x = 6. What is the point after one iteration?
- x = 4
- x = 8
- x = 5.75
- x = −2
Show the answer
x = 4
f′(x) = 2x − 4, so f′(6) = 8. The new point is 6 − 0.25 × 8 = 4. Forgetting the minus sign gives 6 + 2 = 8; forgetting the step gives 6 − 8 = −2; forgetting the derivative gives 6 − 0.25 = 5.75.
At each iteration, the new point is the old one … the step times the … at the old point.
Show the answer
At each iteration, the new point is the old one minus the step times the derivative at the old point.
If the slope is positive, the function rises to the right: we move back, hence the minus sign. If it is negative, we move to the right.