Decision trees and random forests
A classifier that asks threshold questions: reading a tree, measuring impurity (Gini, entropy), choosing the best split, bounding the depth; then the random forest, which makes bootstrapped trees vote.
- 27 cards
- 5 topics
- Free
The outline, topic by topic
Each topic is weighted the way the exam weighs it.
- Concepts
- Questions
- Sort and link
- The numbers
- Scenarios
A few cards from the deck
Decision tree: what is it?
Show the answer
A model that chains threshold questions on the features, each answer leading down a branch, until a leaf that gives the prediction.
It reads as a series of “if … then …” rules, which makes it easy to explain. It is learned by choosing at each node the split that best purifies the data.
A simple tree on the irises first asks “petal length ≤ 2.45 cm?”: if yes, setosa. Otherwise it asks “petal width ≤ 1.75 cm?”: if yes, versicolor, else virginica. Which species does it predict for a petal 4.9 cm long and 1.8 cm wide?
- Virginica: 4.9 > 2.45 then 1.8 > 1.75
- Setosa: petal length is the first feature tested
- Versicolor: 1.8 cm wide stays close to the 1.75 cm threshold
- None: a petal 4.9 cm long matches no leaf
Show the answer
Virginica: 4.9 > 2.45 then 1.8 > 1.75
Go down node by node: 4.9 is not ≤ 2.45, go right; 1.8 is not ≤ 1.75, go right again, into the virginica leaf. Every individual reaches exactly one leaf.
To curb a tree's overfitting, its … or the minimum number of individuals per … is bounded.
Show the answer
To curb a tree's overfitting, its depth or the minimum number of individuals per leaf is bounded.
In scikit-learn, these are
max_depthandmin_samples_leaf. The deeper the tree, the smaller its zones and the more they cling to the training points.Gini impurity: what is it?
Show the answer
A measure of a node's impurity: 1 minus the sum of the squared proportions of each class; it is 0 when the node is pure.
With two classes, its maximum is 0.5 (half and half). A split is good when the weighted average of the two children's impurities is far lower than the parent's.