CF

Streaming with Kafka & Spark

2h 46m 25s
English
Paid
Updated September 2026

Streaming with Kafka & Spark is a 40-lesson 2 hours 46 minutes self-paced course by Andreas Kretz. Andreas Kretz built this course around a single realistic project: processing invoices from an online store in real time and visualizing that data through a working dashboard, using FastAPI, Apache Kafka, Apache Spark, MongoDB, and…

Course facts

Lessons
40
Duration
2 hours 46 minutes
Level
All levels
Language
English
Updated
2026-09-11
Instructor
Andreas Kretz
Price
Premium

Andreas Kretz built this course around a single realistic project: processing invoices from an online store in real time and visualizing that data through a working dashboard, using FastAPI, Apache Kafka, Apache Spark, MongoDB, and Streamlit together.

How the pipeline comes together

  • Load a Kaggle dataset, saved first as CSV and then converted to JSON
  • Design and build a basic API with FastAPI, testing it with Postman
  • Run Kafka and the API as Docker services, with topics configured for the data flow
  • Stream data through Spark into Kafka using Spark Structured Streaming
  • Set up MongoDB and Mongo-Express with Docker for storage
  • Move streaming data from Kafka into MongoDB as nested JSON documents
  • Build a Python API client to send data in for processing
  • Build a Streamlit dashboard to review customer invoices and product details

Across 40 lessons and a companion GitHub repo, Docker experience is genuinely useful going in, since it underpins most of the environment — by the end, you'll have built one coherent streaming system rather than isolated demos of Kafka, Spark, or MongoDB on their own.

Additional

https://github.com/team-data-science/document-streaming

Who teaches Streaming with Kafka & Spark? Andreas Kretz

Andreas Kretz thumbnail

Andreas Kretz is a German data engineer and one of the most widely followed independent voices on data engineering as a career discipline. He runs the Plumbers of Data Science brand and has been publishing tutorial material continuously since the field consolidated around the modern lake-house stack (Spark, Kafka, Snowflake, Databricks, Airflow).

His CourseFlix listing is the largest single-author catalog under this source — over thirty courses spanning data-pipeline construction, streaming architectures, the cloud-native data stack on AWS / Azure / GCP, the Python and Scala tooling that dominates the field, and the soft-skills / career side of breaking into data engineering. Material is paid and aimed at engineers transitioning into data work or already-working data engineers picking up specific tools.

What lessons are included in Streaming with Kafka & Spark?

This is a demo lesson (10:00 remaining)

You can watch up to 10 minutes for free. Subscribe to unlock all 40 lessons in this course and access 10,000+ hours of premium content across all courses.

View Pricing
0:00
/
#1: Introduction
All Course Lessons (40)
#Lesson TitleDurationAccess
1
Introduction Demo
01:13
2
Project overview
05:34
3
Docker Fundamentals
01:44
4
The Dataset we use
02:49
5
Transform CSV to JSONs
10:52
6
API Schema
03:43
7
Creating the API with FastAPI
09:42
8
Testing the API with Postman
06:11
9
Apache Kafka Goals
02:34
10
Kafka Docker Compose Explained
03:36
11
Startup Kafka Compose File
02:47
12
Kafka Topics Setup
07:12
13
Preparing the API Docker build
04:14
14
Build the API
03:26
15
Deploy the API
02:49
16
Test the API Container with Kafka
02:07
17
Recap API & Kafka
01:38
18
Apache Spark Compose Config
04:39
19
Startup Spark with Kafka & API
02:27
20
Spark Ingest Kafka & Produce Kafka
06:35
21
Setup Test configuration
03:02
22
Test Spark Streaming Kafka
05:43
23
Spark UI Monitoring
02:31
24
MongoDB Goals
04:23
25
MongoDB Docker Compose Conifg
03:59
26
MongoDB Startup
02:45
27
Prepare MongoDB Database & Collection
01:46
28
Spark Code Streaming To MongoDB
06:32
29
Transformations 1: Writing Kafka Message as String to MongoDB
03:26
30
Transformations 2: Writing complete Kafka message to MongoDB
02:35
31
Transformations 3: Writing Nested Document to MongoDB
04:29
32
Transformations 4: Writing Messages as Document
02:14
33
Spark Streaming Conclusion
02:53
34
Writing the API Client
04:05
35
Create Test Data & Run Client
05:37
36
Streamlit Intro & Goals
06:19
37
Query Customer Invoices
04:08
38
Query Invoice Documents
04:16
39
Project Summary
03:26
40
Outlook
06:24
Unlock unlimited learning

Get instant access to all 39 lessons in this course, plus thousands of other premium courses. One subscription, unlimited knowledge.

Learn more about subscription

What courses are similar to Streaming with Kafka & Spark?

More courses by Andreas Kretz

Frequently asked questions

What prior knowledge do I need before taking this course?
Before enrolling in this course, you should have a foundational understanding of Docker, as it is integral to the project environment. Additionally, familiarity with FastAPI, Apache Kafka, Apache Spark, and MongoDB is essential, as these technologies are central to the course content and project implementation.
What project will I work on during this course?
The course involves building an end-to-end data processing project for an online store. You will handle real-time data processing, including client invoices, and visualize the data using a user-friendly interface. Key technologies such as FastAPI for API development, Apache Kafka for data streaming, Apache Spark for processing, and MongoDB for storage are used throughout the project.
Who is the target audience for this course?
This course is designed for individuals interested in real-time data processing and visualization, particularly those who have a foundational understanding of the involved technologies. It's suitable for data engineers, software developers, and IT professionals looking to enhance their skills in building and deploying data pipelines using modern tools.
How does this course compare in depth and scope to similar courses?
This course offers a focused and practical approach to real-time data processing with a complete project utilizing Apache Kafka and Spark, unlike some introductory courses. It covers practical implementation details, such as Dockerizing services, setting up Apache Kafka and Spark, and streaming data into a MongoDB database, providing a thorough understanding of the entire data pipeline.
What specific tools and platforms will I learn to use?
You'll learn to use several key tools and platforms: FastAPI for API development, Apache Kafka for real-time data streaming, Apache Spark for data processing, MongoDB for data storage, and Streamlit for data visualization. Additionally, Docker is used extensively to manage the deployment of these services.
What topics are not covered in this course?
While the course covers real-time data processing and related technologies, it does not delve into the basics of each technology, such as introductory courses on Docker, FastAPI, Apache Kafka, or Spark. It also does not cover data science methodologies or detailed data analytics beyond the scope of streaming and visualization.
How much time should I expect to commit to this course?
The course comprises 40 lessons which include hands-on exercises and projects. While the exact runtime is not specified, students should be prepared to invest significant time in both learning and practical application of the concepts. Completing the course will require a commitment to understanding and applying complex real-time data processing techniques.