Skip to content

Big data

Process very large data volumes with Spark and the lakehouse.

Your running project

Track a country's market prices

The need
Bring together years of readings and answer quickly: where and when do prices rise?
The data
Public price records, such as those from the World Food Programme.
External tools
SparkDatabricksSQLGitHub

Workshop

Establish the needs

Who uses it, what problem it solves, how to tell it worked.

Precision of your question: 0/5

What you will learn

  • Python and SQL for large volumes
  • Apache Spark: batch and streaming
  • Distributed storage and lakehouse architecture
  • Reliability, cost and monitoring of data pipelines

Certifications to aim for at the end

No certification is titled “big data”: these, on Spark and the lakehouse, are the closest.

These certifications are issued by their own bodies, not by Nangmo, and their exams are paid for with them. Checked on 3 November 2026: confirm the language, price and exam location on their site before registering. These exams are generally taken without AI: you will need to read and reason on your own.

How to start a career in this field

  1. 1Learn to read Python and SQL, then have Spark process a real dataset with the guide, understanding each step.
  2. 2Publish a complete, documented data pipeline on GitHub.
  3. 3Take the Spark certification, then Data Engineer Associate.
  4. 4Aim for a first job as a junior data engineer or ETL developer, at a bank, telecom operator or consultancy.

First jobs to aim for

  • Junior data engineer
  • ETL developer
  • Spark developer

The workshop is open, but this path's lessons are not online yet. Tell us you are interested: it helps us decide which to write first.