Streaming with Kafka & Spark
2h 46m 25s
English
Paid
Course description
This course is a full-fledged project with a complete cycle of real-time data processing. You will work with data from an online store, including client invoices and goods from these invoices. The aim of the course is to set up streaming data processing (invoices) as they arrive and visualize them in a convenient interface.
You will use technologies such as FastAPI, Apache Kafka, Apache Spark, MongoDB, and Streamlit - tools you are already familiar with from other courses. We strongly recommend completing basic courses on these technologies, as well as a course on Docker fundamentals, since the project heavily relies on a Docker environment.
Read more about the course
What you can expect in the course:
- Introduction to the Project
- You will understand the architecture of an end-to-end pipeline and see how data visualization is built. Step by step, you will learn how to build a project and at what stage to use certain technologies.
- Data Preparation
- You will load and transform a dataset from Kaggle: first saving it in CSV format, then converting the data into JSON for further work.
- API with FastAPI
- You will get acquainted with the general API scheme, create an API with FastAPI, set it up to receive data, and test its operation through Postman.
- Apache Kafka and API as Docker Services
- You will install Apache Kafka through Docker, set up topics, write an API that will send data to Kafka, and deploy it in a Docker container.
- Data Streaming through Spark into Kafka
- You will prepare a container with Apache Spark, connect it to Kafka and the API, set up data processing through Spark Structured Streaming, and test the pipeline.
- Data Storage in MongoDB
- You will set up MongoDB and Mongo-Express through Docker, prepare a database and collection for storing data, and link Spark with MongoDB.
- Data Streaming from Kafka to MongoDB
- You will learn how to work with Spark Structured Streaming to write streaming data from Kafka into MongoDB in the format of nested JSON documents.
- API Client in Python
- You will write a Python client script to send data to the API in JSON format and ensure that the data is successfully recorded in MongoDB.
- Visualization Interface in Streamlit
- You will build an interactive dashboard to view customer invoices and products using Streamlit.
This project is a great opportunity to combine knowledge of API, data streaming processing, working with Docker, and databases into a cohesive whole, gaining practical experience in creating streaming applications in real-world conditions.
Watch Online
Watch Online Streaming with Kafka & Spark
0:00
/ #1: Introduction
All Course Lessons (40)
| # | Lesson Title | Duration | Access |
|---|---|---|---|
| 1 | Introduction Demo | 01:13 | |
| 2 | Project overview | 05:34 | |
| 3 | Docker Fundamentals | 01:44 | |
| 4 | The Dataset we use | 02:49 | |
| 5 | Transform CSV to JSONs | 10:52 | |
| 6 | API Schema | 03:43 | |
| 7 | Creating the API with FastAPI | 09:42 | |
| 8 | Testing the API with Postman | 06:11 | |
| 9 | Apache Kafka Goals | 02:34 | |
| 10 | Kafka Docker Compose Explained | 03:36 | |
| 11 | Startup Kafka Compose File | 02:47 | |
| 12 | Kafka Topics Setup | 07:12 | |
| 13 | Preparing the API Docker build | 04:14 | |
| 14 | Build the API | 03:26 | |
| 15 | Deploy the API | 02:49 | |
| 16 | Test the API Container with Kafka | 02:07 | |
| 17 | Recap API & Kafka | 01:38 | |
| 18 | Apache Spark Compose Config | 04:39 | |
| 19 | Startup Spark with Kafka & API | 02:27 | |
| 20 | Spark Ingest Kafka & Produce Kafka | 06:35 | |
| 21 | Setup Test configuration | 03:02 | |
| 22 | Test Spark Streaming Kafka | 05:43 | |
| 23 | Spark UI Monitoring | 02:31 | |
| 24 | MongoDB Goals | 04:23 | |
| 25 | MongoDB Docker Compose Conifg | 03:59 | |
| 26 | MongoDB Startup | 02:45 | |
| 27 | Prepare MongoDB Database & Collection | 01:46 | |
| 28 | Spark Code Streaming To MongoDB | 06:32 | |
| 29 | Transformations 1: Writing Kafka Message as String to MongoDB | 03:26 | |
| 30 | Transformations 2: Writing complete Kafka message to MongoDB | 02:35 | |
| 31 | Transformations 3: Writing Nested Document to MongoDB | 04:29 | |
| 32 | Transformations 4: Writing Messages as Document | 02:14 | |
| 33 | Spark Streaming Conclusion | 02:53 | |
| 34 | Writing the API Client | 04:05 | |
| 35 | Create Test Data & Run Client | 05:37 | |
| 36 | Streamlit Intro & Goals | 06:19 | |
| 37 | Query Customer Invoices | 04:08 | |
| 38 | Query Invoice Documents | 04:16 | |
| 39 | Project Summary | 03:26 | |
| 40 | Outlook | 06:24 |
Unlock unlimited learning
Get instant access to all 39 lessons in this course, plus thousands of other premium courses. One subscription, unlimited knowledge.
Learn more about subscriptionComments
0 commentsSimilar courses
Machine Learning in JavaScript with TensorFlow.js
Sources: udemy
Interested in using Machine Learning in JavaScript applications and websites? Then this course is for you! This is the tutorial you've been looking for to becom
6 hours 42 minutes 20 seconds
Build a Large Language Model (From Scratch)
Sources: Sebastian Raschka
"Creating a Large Language Model from Scratch" is a practical guide that will teach you step by step how to create, train, and fine-tune large language models..
Data Structures and Algorithmic Trading: Machine Learning
Sources: udemy
Data Structures and Algorithmic trading is a method of executing orders using automated pre-programmed trading instructions over time. They were developed so th
2 hours 20 minutes 32 seconds
dbt for Data Engineers
Sources: Andreas Kretz
dbt (data build tool) is a data transformation tool with a priority on SQL. It allows for simple and transparent transformation, testing, and documentation...
1 hour 52 minutes 55 seconds
PyTorch for Deep Learning
Sources: zerotomastery.io
Master PyTorch for deep learning with a step-by-step course. Build real-world projects and enhance your skills to become a Deep Learning Engineer.
52 hours 27 seconds
Want to join the conversation?
Sign in to comment