Fundamentals of Apache Airflow
Course description
This practical course starts with the basics and step by step guides you to building real orchestration scenarios - from task retries to integration with Spark and loading external data.
Moving data from point A to point B is only a small part of the task. It is important that data is delivered accurately, reliably, and automatically - and this is where Apache Airflow comes to the rescue.
You will learn how to transform chaotic, manually configured pipelines into well-organized workflows. We'll begin with understanding the architecture of Airflow and its key components, followed by mastering more advanced techniques: setting up retries, handling failures, using sensors, working with Apache Spark, and automatically loading data from external sources into a data lake.
The course is suitable for both beginner data engineers and those who want to improve their orchestration skills. You will receive real practical tools for creating scalable and reliable data processing systems.
Watch Online
Watch Online Fundamentals of Apache Airflow
All Course Lessons (27)
# | Lesson Title | Duration | Access |
---|---|---|---|
1 | Introduction Demo | 07:20 | |
2 | What Is Apache Airflow? | 05:19 | |
3 | Airflow’s Architecture | 03:15 | |
4 | [Optional] What Is a Virtualenv? | 06:37 | |
5 | [Optional] What Is Docker? | 11:03 | |
6 | Installing Spark | 05:51 | |
7 | Installing Airflow | 06:33 | |
8 | Defining an Airflow DAG | 08:03 | |
9 | Errors Handling | 03:38 | |
10 | Idempotent Tasks | 04:54 | |
11 | Creating a DAG - Part 1 | 04:58 | |
12 | Creating a DAG - Part 2 | 04:42 | |
13 | Handling Failed Tasks | 04:09 | |
14 | [Exercise] Data Validation | 04:31 | |
15 | [Exercise] Data Validation - Solution | 03:27 | |
16 | Spark with Airflow | 03:02 | |
17 | Using Spark with Airflow - Part 1 | 07:39 | |
18 | Using Spark with Airflow - Part 2 | 05:52 | |
19 | Sensors In Airflow | 04:46 | |
20 | Using File Sensors | 04:08 | |
21 | Data Ingestion | 05:50 | |
22 | Reading Data From Postgres - Part 1 | 06:03 | |
23 | Reading Data from Postgres - Part 2 | 05:40 | |
24 | [Exercise] Average Customer Review | 03:53 | |
25 | [Exercise] Average Customer Review - Solution | 04:33 | |
26 | Advanced DAGs | 04:26 | |
27 | Let's Keep Learning Together! | 01:06 |
Unlock unlimited learning
Get instant access to all 26 lessons in this course, plus thousands of other premium courses. One subscription, unlimited knowledge.
Learn more about subscriptionComments
0 commentsSimilar courses

Machine Learning with Javascript

PyTorch for Deep Learning with Python Bootcamp

dbt for Data Engineers

Case Study in Product Data Science

Want to join the conversation?
Sign in to comment