Collecting big data and its file formats
Where big data comes from and how it is stored: the three Vs, batch or stream collection, API pagination, and six file formats (CSV, JSON Lines, XML, Avro, Parquet, ORC) compared.
- 29 cards
- 7 topics
- Free
The outline, topic by topic
Each topic is weighted the way the exam weighs it.
- Concepts
- Questions
- Sort and link
- The numbers
- On the diagram
- Scenarios
- Understand
A few cards from the deck
The three Vs of big data: what is it?
Show the answer
Volume, velocity and variety: how much data there is, how fast it arrives and how varied its forms are.
A dataset becomes "big" when one of the three criteria exceeds what a classic tool on a single machine can handle.
A CSV file contains the line below, with "," as separator. How many columns does a CSV reader find in it?
```csv
7,"Dupont, Léa",Lyon,42.5`- 4
- 5
- 3
- 6
Show the answer
4
The comma inside the quotes belongs to the value "Dupont, Léa": it separates nothing. Without quotes the line would have five columns and everything would shift.
Common sources of big data are server …, …, APIs and open data.
Show the answer
Common sources of big data are server logs, sensors, APIs and open data.
Each source sets its own pace and format: a log is text appended line by line, a sensor emits regular readings, an API returns paginated JSON.