Fundamentals of Apache Airflow

2h 21m 18s
English
Paid

Course description

This practical course starts with the basics and step by step guides you to building real orchestration scenarios - from task retries to integration with Spark and loading external data.

Moving data from point A to point B is only a small part of the task. It is important that data is delivered accurately, reliably, and automatically - and this is where Apache Airflow comes to the rescue.

You will learn how to transform chaotic, manually configured pipelines into well-organized workflows. We'll begin with understanding the architecture of Airflow and its key components, followed by mastering more advanced techniques: setting up retries, handling failures, using sensors, working with Apache Spark, and automatically loading data from external sources into a data lake.

The course is suitable for both beginner data engineers and those who want to improve their orchestration skills. You will receive real practical tools for creating scalable and reliable data processing systems.

Watch Online

This is a demo lesson (10:00 remaining)

You can watch up to 10 minutes for free. Subscribe to unlock all 27 lessons in this course and access 10,000+ hours of premium content across all courses.

View Pricing
0:00
/
#1: Introduction

All Course Lessons (27)

#Lesson TitleDurationAccess
1
Introduction Demo
07:20
2
What Is Apache Airflow?
05:19
3
Airflow’s Architecture
03:15
4
[Optional] What Is a Virtualenv?
06:37
5
[Optional] What Is Docker?
11:03
6
Installing Spark
05:51
7
Installing Airflow
06:33
8
Defining an Airflow DAG
08:03
9
Errors Handling
03:38
10
Idempotent Tasks
04:54
11
Creating a DAG - Part 1
04:58
12
Creating a DAG - Part 2
04:42
13
Handling Failed Tasks
04:09
14
[Exercise] Data Validation
04:31
15
[Exercise] Data Validation - Solution
03:27
16
Spark with Airflow
03:02
17
Using Spark with Airflow - Part 1
07:39
18
Using Spark with Airflow - Part 2
05:52
19
Sensors In Airflow
04:46
20
Using File Sensors
04:08
21
Data Ingestion
05:50
22
Reading Data From Postgres - Part 1
06:03
23
Reading Data from Postgres - Part 2
05:40
24
[Exercise] Average Customer Review
03:53
25
[Exercise] Average Customer Review - Solution
04:33
26
Advanced DAGs
04:26
27
Let's Keep Learning Together!
01:06

Unlock unlimited learning

Get instant access to all 26 lessons in this course, plus thousands of other premium courses. One subscription, unlimited knowledge.

Learn more about subscription

Comments

0 comments

Want to join the conversation?

Sign in to comment

Similar courses

Schema Design Data Stores

Schema Design Data Stores

Sources: Andreas Kretz
During my coaching sessions, one important topic repeatedly comes up - designing diagrams. Therefore, I decided to create a separate course in the academy to...
2 hours 30 minutes 25 seconds
PyTorch for Deep Learning with Python Bootcamp

PyTorch for Deep Learning with Python Bootcamp

Sources: udemy
Welcome to the best online course for learning about Deep Learning with Python and PyTorch! PyTorch is an open source deep learning platform that provides a sea
17 hours 2 minutes 14 seconds
DS4B 101-P: Python for Data Science Automation

DS4B 101-P: Python for Data Science Automation

Sources: Business Science University
Python for Data Science Automation is an innovative course designed to teach data analysts how to convert business processes to python-based data science automations. The course...
27 hours 6 minutes 1 second
Data Engineering on AWS

Data Engineering on AWS

Sources: Andreas Kretz
This course is the perfect start for those who want to learn cloud technologies and start working with Amazon Web Services (AWS), one of the most popular..
4 hours 46 minutes 38 seconds
Azure Data Pipelines with Terraform

Azure Data Pipelines with Terraform

Sources: Andreas Kretz
Azure is becoming an increasingly popular platform for companies using the Microsoft365 ecosystem. If you want to enhance your data engineering skills...
4 hours 20 minutes 29 seconds