Exploring a massive dataset
The fixed plan of a first exploration: dimensions and types, per-column profile, sampling, group aggregates and a trend that reverses across subgroups.
- 21 cards
- 6 topics
- Free
The outline, topic by topic
Each topic is weighted the way the exam weighs it.
- Concepts
- Questions
- Sort and link
- The numbers
- Scenarios
- Understand
A few cards from the deck
Exploratory data analysis: what is it?
Show the answer
A first reading of a dataset through summaries and charts, to grasp its shape and spot oddities before any modelling.
You start with dimensions and types, then the profile of each column, then crossings: it is a plan, not a random walk.
The table df has 10 rows: 4 for Lyon (10, 20, 30, 40), 3 for Paris (40, 60, 20) and 3 for Nice (5, 15, 25). What is the Lyon line of this code?
```python
print(df.groupby("ville")["montant"].mean())`25.0100.010.040.0
Show the answer
25.0Lyon's mean is (10 + 20 + 30 + 40) / 4 = 25.0. 100 would be the sum (
sum()), 10 the minimum (min()) and 40 the maximum (max()), which is also Paris's mean.A column's profile summarises its …, its share of … values, its range (minimum and maximum) and its distribution.
Show the answer
A column's profile summarises its cardinality, its share of missing values, its range (minimum and maximum) and its distribution.
A maximum of 9999 used as a default value, 62% blanks or an identifier read as a number show up in this profile, before any computation.
Column cardinality: what is it?
Show the answer
The number of distinct values a column contains.
A cardinality of 1 flags a constant, useless column; a cardinality equal to the number of rows flags an identifier, not to be treated as a measurement.