Big data
Process very large data volumes with Spark and the lakehouse.
Your running project
Track a country's market prices
- The need
- Bring together years of readings and answer quickly: where and when do prices rise?
- The data
- Public price records, such as those from the World Food Programme.
- External tools
- SparkDatabricksSQLGitHub
Workshop
Establish the needs
Who uses it, what problem it solves, how to tell it worked.
What you will learn
- Python and SQL for large volumes
- Apache Spark: batch and streaming
- Distributed storage and lakehouse architecture
- Reliability, cost and monitoring of data pipelines
Certifications to aim for at the end
No certification is titled “big data”: these, on Spark and the lakehouse, are the closest.
- Entry levelDatabricks Certified Associate Developer for Apache SparkDatabricks
- IntermediateDatabricks Certified Data Engineer AssociateDatabricks
- AdvancedDatabricks Certified Data Engineer ProfessionalDatabricks
These certifications are issued by their own bodies, not by Nangmo, and their exams are paid for with them. Checked on 3 November 2026: confirm the language, price and exam location on their site before registering. These exams are generally taken without AI: you will need to read and reason on your own.
How to start a career in this field
- 1Learn to read Python and SQL, then have Spark process a real dataset with the guide, understanding each step.
- 2Publish a complete, documented data pipeline on GitHub.
- 3Take the Spark certification, then Data Engineer Associate.
- 4Aim for a first job as a junior data engineer or ETL developer, at a bank, telecom operator or consultancy.
First jobs to aim for
- Junior data engineer
- ETL developer
- Spark developer
The workshop is open, but this path's lessons are not online yet. Tell us you are interested: it helps us decide which to write first.