Data cleaning
Six defects of a dataset (missing values, duplicates, outliers, inconsistent formats, wrong types, inconsistent columns) and how to treat them, keeping a trace of what was changed.
- 25 cards
- 6 topics
- Free
The outline, topic by topic
Each topic is weighted the way the exam weighs it.
- Concepts
- Questions
- Sort and link
- The numbers
- Scenarios
- Understand
A few cards from the deck
Data cleaning: what is it?
Show the answer
The step that detects, then fixes or removes, the wrong, absent or inconsistent values of a dataset before analysing it.
An analysis, however well done, is only as good as its data: uncorrected errors end up in averages, models and charts.
The table df has 10 rows and 2 columns. Rows 3, 4 and 8 each contain a blank and match no other row; rows 0 and 2 are identical, as are rows 6 and 7. What does this code print?
```python
print(df.dropna().shape)
print(df.duplicated().sum())`(7, 2)then2(7, 2)then4(10, 2)then2(8, 2)then2
Show the answer
(7, 2)then2dropna()removes the 3 rows with blanks and leaves 7 rows of 2 columns.duplicated()flags the second occurrence of each duplicate: 2 flags, not 4, because the first occurrence is not counted.The six classic defects of a dataset are … values, duplicates, outliers, inconsistent formats, wrong types and inconsistencies between ….
Show the answer
The six classic defects of a dataset are missing values, duplicates, outliers, inconsistent formats, wrong types and inconsistencies between columns.
An end date before the start date is an inconsistency between columns: each cell looks valid alone, only comparing them reveals the error.
Missing value: what is it?
Show the answer
A cell of a dataset that holds no measurement, shown as blank, NaN, NULL or a code like "N/A".
You can drop the row, impute a value or keep it as is: the right choice depends on the share of blanks and why the value is absent.