CF

Apache Kafka Fundamentals

1h 4m 52s
English
Paid

Master the fundamentals of Apache Kafka in this comprehensive course designed to provide you with essential knowledge for a confident start. You will learn to configure a message queue, write producers and consumers, and understand Kafka's role within data and event processing architectures.

Understanding Kafka and Message Queues

Discover what Kafka is and its usage in stream and event processing systems. You will gain insights into the key components of Kafka, including topics, messages, and consumer groups. Learn how these components interact, how data is written and read from a message queue, and the significance of message order and delivery guarantees.

Exploring Apache Kafka Architecture

Dive deep into Kafka's architecture. Understand topic partitions and their relation to brokers. You will explore data processing within Kafka and learn about Zookeeper, its roles, and its interactions with Kafka brokers and metadata.

Setting Up Your Kafka Development Environment

Learn to run Kafka on a Windows environment using Docker. This section includes a step-by-step guide on setting up a Bitnami Kafka Docker container, complete with practical tips for successful installation and environment launch.

Hands-On Practice with Kafka

Set up your own Kafka topic and master the basic commands to manage it. You will create a producer to write messages and a consumer to read them. Test their functionality using Python and manage consumer offsets utilizing the offset checker.

Integrating Kafka into Data Processing Platforms

Conclude the course by exploring Kafka's integration into Data Science platforms. Examine three practical scenarios of using Kafka:

  • ETL ingest pipeline
  • Multiple consumer processes
  • Multistage stream processing

These examples will equip you with the knowledge to implement Kafka in your everyday work effectively.

Additional

https://hub.docker.com/r/bitnami/kafka

https://github.com/team-data-science/apache-kafka

About the Author: Andreas Kretz

Andreas Kretz thumbnail

Andreas Kretz is a German data engineer and one of the most widely followed independent voices on data engineering as a career discipline. He runs the Plumbers of Data Science brand and has been publishing tutorial material continuously since the field consolidated around the modern lake-house stack (Spark, Kafka, Snowflake, Databricks, Airflow).

His CourseFlix listing is the largest single-author catalog under this source — over thirty courses spanning data-pipeline construction, streaming architectures, the cloud-native data stack on AWS / Azure / GCP, the Python and Scala tooling that dominates the field, and the soft-skills / career side of breaking into data engineering. Material is paid and aimed at engineers transitioning into data work or already-working data engineers picking up specific tools.

Watch Online 15 lessons

This is a demo lesson (10:00 remaining)

You can watch up to 10 minutes for free. Subscribe to unlock all 15 lessons in this course and access 10,000+ hours of premium content across all courses.

View Pricing
0:00
/
#1: Introduction
All Course Lessons (15)
#Lesson TitleDurationAccess
1
Introduction Demo
02:16
2
What is Kafka
09:15
3
Basic Kafka Parts
04:25
4
Message Queue Basics
07:39
5
Topics Partitions & Brokers
02:16
6
Brokers & Zookeeper
04:40
7
Development Environment
02:42
8
Bitnami Docker Setup
03:34
9
Basic Topic Commands
04:16
10
Kafka Producer
05:56
11
Kafka Consumer
01:35
12
Testing Producer & Consumer
02:45
13
Working with Consumer Offsets
06:46
14
Examples How Kafka Fits in Data Platforms
05:25
15
Conclusion
01:22
Unlock unlimited learning

Get instant access to all 14 lessons in this course, plus thousands of other premium courses. One subscription, unlimited knowledge.

Learn more about subscription

Related courses

Frequently asked questions

What prerequisites should I have before taking this course?
Prior knowledge of distributed systems or experience with message queues can be beneficial but is not mandatory. Familiarity with basic programming concepts and experience in Python is recommended, as the course includes practical exercises using Python to test Kafka producers and consumers.
What kind of projects or skills will I be able to build by the end of the course?
By the end of the course, you will be able to configure a message queue, set up and manage Kafka topics, and write producers and consumers. You will also learn to run Kafka in a Windows environment using Docker and perform hands-on practice to test functionality using Python.
Who is the target audience for this course?
This course is ideal for software developers, data engineers, and system architects who are new to Apache Kafka and want to understand its fundamentals and role within data and event processing architectures. It is designed for those looking to integrate Kafka into data platforms.
How does the scope of this course compare to other Kafka courses?
The course focuses on foundational aspects of Apache Kafka, such as understanding Kafka's architecture, setting up a development environment, and basic commands for producers and consumers. It emphasizes practical skills using Kafka in development environments, which may not be covered in more advanced courses that focus on Kafka's integration with other systems.
What specific tools and platforms are covered in the course?
The course covers tools like Bitnami Docker for setting up a Kafka development environment on Windows. It also involves practical exercises using Python to test Kafka producers and consumers, and discusses the role of Zookeeper in managing Kafka brokers and metadata.
What topics are explicitly not covered in this course?
This course does not cover advanced Kafka topics such as stream processing with Kafka Streams, integrating Kafka with other big data tools, or detailed performance tuning. It is focused on the fundamental concepts and basic setup and operation of Kafka.
How much time should I expect to commit to this course?
The course consists of 15 lessons, each designed to provide a thorough understanding of Apache Kafka fundamentals. While exact runtime details are not provided, students should allocate sufficient time for both lesson review and practical exercises, especially if setting up and testing the environment on their own systems.