Streaming with Kafka & Spark

What you can expect in the course:

• Introduction to the Project
  • You will understand the architecture of an end-to-end pipeline and see how data visualization is built. Step by step, you will learn how to build a project and at what stage to use certain technologies.
• Data Preparation
  • You will load and transform a dataset from Kaggle: first saving it in CSV format, then converting the data into JSON for further work.
• API with FastAPI
  • You will get acquainted with the general API scheme, create an API with FastAPI, set it up to receive data, and test its operation through Postman.
• Apache Kafka and API as Docker Services
  • You will install Apache Kafka through Docker, set up topics, write an API that will send data to Kafka, and deploy it in a Docker container.
• Data Streaming through Spark into Kafka
  • You will prepare a container with Apache Spark, connect it to Kafka and the API, set up data processing through Spark Structured Streaming, and test the pipeline.
• Data Storage in MongoDB
  • You will set up MongoDB and Mongo-Express through Docker, prepare a database and collection for storing data, and link Spark with MongoDB.
• Data Streaming from Kafka to MongoDB
  • You will learn how to work with Spark Structured Streaming to write streaming data from Kafka into MongoDB in the format of nested JSON documents.
• API Client in Python
  • You will write a Python client script to send data to the API in JSON format and ensure that the data is successfully recorded in MongoDB.
• Visualization Interface in Streamlit
  • You will build an interactive dashboard to view customer invoices and products using Streamlit.

This project is a great opportunity to combine knowledge of API, data streaming processing, working with Docker, and databases into a cohesive whole, gaining practical experience in creating streaming applications in real-world conditions.