Apache Kafka Series - Learn Apache Kafka for Beginners v3
Welcome to the Apache Kafka Series! Join a community of 20,000+ students learning Kafka. Apache Kafka has become the leading distributed data streaming enterprise big data technology. Kafka is used in production by over 33% of the Fortune 500 companies such as Netflix, Airbnb, Uber, Walmart and LinkedIn. To learn Kafka easily, step-by-step, you have come to the right place! No prior Kafka knowledge is required
More
If you look at the documentation, you can see that Apache Kafka is not easy to learn...
Thanks to my several years of experience in Kafka and Big Data, I wanted to make learning Kafka accessible to everyone.
We'll take a step-by-step approach to learn all the fundamentals of Apache Kafka.
At the end of this course, you'll be productive and you'll know the following:
The Apache Kafka Ecosystem Architecture
The Kafka Core Concepts: Topics, Partitions, Brokers, Replicas, Producers, Consumers, and more!
Launch your own Kafka cluster in no time using native Kafka binaries – Windows / MacOS X / Linux
Learn and Practice using the Kafka Command Line Interface (CLI)
Code Producer and Consumers using the Java API
Real world project using Twitter as a source of data for a producer and ElasticSearch as a sink for our consumer
Overview of Advanced APIs (Kafka Connect, Kafka Streams)
Real World Case Studies and Big Use cases
Overview of Advanced Kafka for Administrators
Advanced Topic Configurations
Annexes (starting a Kafka cluster locally, using Docker, etc...)
Note: The hands-on section is based on Java, which is the native Kafka programming language. But, good news! Your learning in Java will be completely applicable to other programming languages, such as Python, C#, Node.js or Scala, and Big Data frameworks such as Spark, NiFi or Akka
This is the course that could improve your career!
Apache Kafka is a skill in high demand and there are not enough people to fulfil all the open positions. You can boost your income, take on new roles and fun challenges. Many of my students are now the Kafka experts of their companies!
I hope to see you inside the course!
Note: Looking for more advanced Kafka concepts? There are many volumes in the Apache Kafka Series:
Learn Kafka for Beginners v2 (this course - great to start)
Kafka Connect Hands On Learning
Kafka Streams for Data Processing
Kafka Cluster Setup & Administration
Confluent Schema Registry & Kafka REST Proxy
Kafka Security (SSL SASL ACL)
Kafka Monitoring and Operations
Happy learning!
- A recent Windows / Mac / Linux machine with minimum 4GB of RAM, 5 GB of disk space
- Some understanding of Java Programming
- Good to have knowledge about Linux command line
- Desire to learn something awesome and new!
- Developers who want to learn the Apache Kafka Fundamentals, start a cluster and write their first application
- Architects who want to understand how Apache Kafka fits into their solution architecture
- Anyone looking to learn the full theory of how Apache Kafka works as a distributed system
What you'll learn:
- Understand Apache Kafka Ecosystem, Architecture, Core Concepts and Operations
- Master Concepts such as Topics, Partitions, Brokers, Producers, Consumers
- Start a personal Kafka development environment
- Learn major CLIs: kafka-topics, kafka-console-producer, kafka-console-consumer, kafka-consumer-groups, kafka-configs
- Create your Producers and Consumers in Java to interact with Kafka
- Program a Real World Twitter Producer & ElasticSearch Consumer
- Extended APIs Overview (Kafka Connect, Kafka Streams), Case Studies and Big Data Architecture
- Practice and Understand Log Compaction
Watch Online Apache Kafka Series - Learn Apache Kafka for Beginners v3
| # | Title | Duration |
|---|---|---|
| 1 | Course Introduction | 02:23 |
| 2 | Apache Kafka in 5 minutes | 05:20 |
| 3 | Course Objectives | 03:58 |
| 4 | Welcome! - About your instructor | 02:46 |
| 5 | Topics, Partitions and Offsets | 06:11 |
| 6 | Producers and Message Keys | 07:25 |
| 7 | Consumers & Deserialization | 04:02 |
| 8 | Consumer Groups & Consumer Offsets | 07:05 |
| 9 | Brokers and Topics | 04:29 |
| 10 | Topic Replication | 05:31 |
| 11 | Producer Acknowledgements & Topic Durability | 02:10 |
| 12 | Zookeeper | 05:15 |
| 13 | Kafka KRaft - Removing Zookeeper | 01:35 |
| 14 | Theory Roundup | 01:34 |
| 15 | Important: Starting Kafka & Lectures Order | 04:07 |
| 16 | Starting Kafka with Conduktor - Multi Platform | 02:11 |
| 17 | Mac OS X - Download and Setup Kafka in PATH | 06:38 |
| 18 | Mac OS X - Start Zookeeper and Kafka | 03:50 |
| 19 | Mac OS X - Using brew | 04:09 |
| 20 | Linux - Download and Setup Kafka in PATH | 07:33 |
| 21 | Linux - Start Zookeeper and Kafka | 03:47 |
| 22 | Windows WSL2 - Download Kafka and PATH Setup | 08:04 |
| 23 | Windows WSL2 - Start Zookeeper & Kafka | 03:21 |
| 24 | Windows WSL2 - How to Fix Problems | 05:28 |
| 25 | Windows non-WSL2 - Start Zookeeper and Kafka | 08:32 |
| 26 | Mac OS X - Start Kafka in KRaft mode | 03:47 |
| 27 | Linux - Start Kafka in KRaft mode | 03:13 |
| 28 | Windows WSL2 - Start Kafka in KRaft mode | 03:05 |
| 29 | CLI Introduction | 03:03 |
| 30 | Kafka Topics CLI | 07:16 |
| 31 | Kafka Console Producer CLI | 06:54 |
| 32 | Kafka Console Consumer CLI | 05:58 |
| 33 | Kafka Consumers in Group | 08:05 |
| 34 | Kafka Consumer Groups CLI | 06:09 |
| 35 | Resetting Offsets | 04:17 |
| 36 | Conduktor - Demo | 05:02 |
| 37 | Kafka SDK List | 01:15 |
| 38 | Creating Kafka Project | 08:38 |
| 39 | Java Producer | 11:06 |
| 40 | Java Producer Callbacks | 12:19 |
| 41 | Java Producer with Keys | 04:40 |
| 42 | Java Consumer | 12:15 |
| 43 | Java Consumer - Graceful Shutdown | 06:56 |
| 44 | Java Consumer inside Consumer Group | 06:02 |
| 45 | Java Consumer Incremental Cooperative Rebalance & Static Group Membership | 07:14 |
| 46 | Java Consumer Incremental Cooperative Rebalance - Practice | 04:54 |
| 47 | Java Consumer Auto Offset Commit Behavior | 03:18 |
| 48 | Programming - Advanced Tutorials | 01:37 |
| 49 | Real World Project Overview | 01:59 |
| 50 | Wikimedia Producer Project Setup | 06:26 |
| 51 | Wikimedia Producer Implementation | 11:55 |
| 52 | Wikimedia Producer Run | 04:45 |
| 53 | Wikimedia Producer - Producer Config Intros | 00:46 |
| 54 | Producer Acknowledgements Deep Dive | 08:49 |
| 55 | Producer Retries | 03:04 |
| 56 | Idempotent Producer | 02:53 |
| 57 | Safe Kafka Producer Settings | 02:00 |
| 58 | Wikimedia Producer Safe Producer Implementation | 04:05 |
| 59 | Kafka Message Compression | 04:49 |
| 60 | linger.ms and batch.size Producer settings | 03:24 |
| 61 | Wikimedia Producer High Throughput Implementation | 03:18 |
| 62 | Producer Default Partitioner & Sticky Partitioner | 04:19 |
| 63 | [Advanced] max.block.ms and buffer.memory | 02:40 |
| 64 | OpenSearch Consumer - Project Overview | 00:53 |
| 65 | OpenSearch Consumer - Project Setup | 03:33 |
| 66 | Setting up OpenSearch on Docker | 02:39 |
| 67 | Setting up OpenSearch on the Cloud | 01:58 |
| 68 | OpenSearch 101 | 04:27 |
| 69 | OpenSearch Consumer Implementation - Part 1 | 07:27 |
| 70 | OpenSearch Consumer Implementation Part 2 | 10:06 |
| 71 | Consumer Delivery Semantics | 03:18 |
| 72 | OpenSearch Consumer Implementation Part 3 - Idempotence | 05:55 |
| 73 | Consumer Offsets Commit Strategies | 04:39 |
| 74 | OpenSearch Consumer Implementation Part 3 - Delivery Semantics | 04:58 |
| 75 | OpenSearch Consumer Implementation Part 5 - Batching Data | 04:20 |
| 76 | Consumer Offset Reset Behavior | 02:07 |
| 77 | OpenSearch Consumer Implementation Part 6 - Replaying Data | 02:13 |
| 78 | Consumer Internal Threads | 05:01 |
| 79 | Consumer Replica Fetching - Rack Awareness | 02:51 |
| 80 | Kafka Extended APIs - Overview | 01:24 |
| 81 | Kafka Connect Introduction | 02:33 |
| 82 | Kafka Connect Wikimedia & ElasticSearch Hands On | 10:58 |
| 83 | Kafka Streams Introduction | 01:53 |
| 84 | Kafka Streams Hands-On | 05:09 |
| 85 | Kafka Schema Registry Introduction | 04:29 |
| 86 | Kafka Schema Registry Hands On | 07:14 |
| 87 | Which Kafka API should I use? | 01:28 |
| 88 | Choosing Partition Count & Replication Factor | 05:22 |
| 89 | Kafka Topics Naming Convention | 01:32 |
| 90 | Case Study - MovieFlix | 05:11 |
| 91 | Case Study - GetTaxi | 04:19 |
| 92 | Case Study - MySocialMedia | 05:33 |
| 93 | Case Study - MyBank | 03:42 |
| 94 | Case Study - Big Data Ingestion | 01:37 |
| 95 | Case Study - Logging and Metrics Aggregation | 01:09 |
| 96 | Kafka Cluster Setup High Level Architecture Overview | 02:56 |
| 97 | Kafka Monitoring & Operations | 02:40 |
| 98 | Kafka Security | 05:52 |
| 99 | Kafka Multi Cluster & MirrorMaker | 04:21 |
| 100 | Advertised Listeners: Kafka Client & Server Communication Protocol | 03:57 |
| 101 | Changing a Topic Configuration | 04:37 |
| 102 | Segment and Indexes | 04:05 |
| 103 | Log Cleanup Policies | 02:54 |
| 104 | Log Cleanup Delete | 02:29 |
| 105 | Log Compaction Theory | 04:53 |
| 106 | Log Compaction Practice | 04:50 |
| 107 | Unclean Leader Election | 01:44 |
| 108 | Large Messages in Kafka | 02:50 |
| 109 | What's Next? | 01:27 |
| 110 | THANK YOU! | 01:33 |
Similar courses to Apache Kafka Series - Learn Apache Kafka for Beginners v3
The Complete 2023 Web Development Bootcamp
Database Design & Implementation
Complete ASP.NET MVC 5
Web API From Start to Finish
Fundamentals of Backend Communications and Protocols
Apache Flink
Fundamentals of Database Engineering
Build an app with ASPNET Core and Angular from scratch
