CF

Fundamentals of Apache Airflow

2h 21m 18s
English
Paid

Fundamentals of Apache Airflow is a 27-lesson 2 hours 21 minutes self-paced course by Zero To Mastery. Enhance your data orchestration skills with our practical course on Apache Airflow .

Course facts

Lessons
27
Duration
2 hours 21 minutes
Level
All levels
Language
English
Updated
Instructor
Zero To Mastery
Price
Premium

Enhance your data orchestration skills with our practical course on Apache Airflow. Begin your journey from the basics and progress towards building real-world orchestration scenarios, including task retries, integration with Spark, and loading external data.

Why Apache Airflow?

While moving data from point A to point B is crucial, ensuring that data is delivered accurately, reliably, and automatically is where Apache Airflow excels. This course will demonstrate how Airflow can transform chaotic, manually configured pipelines into well-organized workflows.

Course Curriculum

Understanding Apache Airflow Architecture

Start with a detailed understanding of the architecture of Airflow and its key components. Lay a solid foundation to build on more advanced concepts.

Advanced Techniques and Features

Master critical techniques such as:

  • Setting up retries to ensure task completion
  • Handling failures gracefully to maintain workflow integrity
  • Utilizing sensors for effective monitoring and control
  • Working with Apache Spark for enhanced data processing
  • Automatically loading data from external sources into a data lake

Who Should Enroll?

This course is perfectly suited for:

  • Beginner data engineers seeking foundational knowledge
  • Experienced professionals aiming to refine their orchestration skills

Equip yourself with practical tools to create scalable and reliable data processing systems using Apache Airflow.

Additional

https://github.com/mushketyk/ztm-data-engineering/tree/main/04-orchestration-with-airflow

Who teaches Fundamentals of Apache Airflow? Zero To Mastery

Zero To Mastery thumbnail

Zero To Mastery (ZTM) is a Toronto-based online coding academy founded by Andrei Neagoie, originally a senior developer at large Canadian tech firms before turning to teaching full-time. The academy's signature is the cohort-based bootcamp track combined with a deep self-paced course library, all aimed at career-changers and self-taught developers preparing to land software-engineering roles at top companies.

The instructor roster has grown well beyond Andrei to include other senior practitioners: Daniel Bourke (machine learning), Aleksa Tešić (DevOps), Jacinto Wong, and others. Courses cover the full software-engineering career path: web development with React and Next.js, Python, machine learning and deep learning, DevOps and cloud, system design, mobile, and the algorithm / data-structure interview prep that gates engineering jobs.

The CourseFlix listing under this source carries over 120 ZTM courses spanning that full range. Material is paid; ZTM itself runs on a monthly / annual membership model. The teaching style favours long-form, project-based courses where students build complete portfolio-quality applications rather than disconnected feature tutorials.

What lessons are included in Fundamentals of Apache Airflow?

This is a demo lesson (10:00 remaining)

You can watch up to 10 minutes for free. Subscribe to unlock all 27 lessons in this course and access 10,000+ hours of premium content across all courses.

View Pricing
0:00
/
#1: Introduction
All Course Lessons (27)
#Lesson TitleDurationAccess
1
Introduction Demo
07:20
2
What Is Apache Airflow?
05:19
3
Airflow’s Architecture
03:15
4
[Optional] What Is a Virtualenv?
06:37
5
[Optional] What Is Docker?
11:03
6
Installing Spark
05:51
7
Installing Airflow
06:33
8
Defining an Airflow DAG
08:03
9
Errors Handling
03:38
10
Idempotent Tasks
04:54
11
Creating a DAG - Part 1
04:58
12
Creating a DAG - Part 2
04:42
13
Handling Failed Tasks
04:09
14
[Exercise] Data Validation
04:31
15
[Exercise] Data Validation - Solution
03:27
16
Spark with Airflow
03:02
17
Using Spark with Airflow - Part 1
07:39
18
Using Spark with Airflow - Part 2
05:52
19
Sensors In Airflow
04:46
20
Using File Sensors
04:08
21
Data Ingestion
05:50
22
Reading Data From Postgres - Part 1
06:03
23
Reading Data from Postgres - Part 2
05:40
24
[Exercise] Average Customer Review
03:53
25
[Exercise] Average Customer Review - Solution
04:33
26
Advanced DAGs
04:26
27
Let's Keep Learning Together!
01:06
Unlock unlimited learning

Get instant access to all 26 lessons in this course, plus thousands of other premium courses. One subscription, unlimited knowledge.

Learn more about subscription

What courses are similar to Fundamentals of Apache Airflow?

Frequently asked questions

What prerequisites are needed before taking this course?
The course is designed for both beginner data engineers and experienced professionals. While there are no strict prerequisites, familiarity with basic programming concepts and some understanding of data engineering principles can be helpful. Optional lessons on Virtualenv and Docker are provided, which can be beneficial if you plan to run Apache Airflow in containerized environments.
What projects or tasks will I build in this course?
Throughout the course, you will build various data orchestration scenarios using Apache Airflow. Key projects include defining and creating Airflow DAGs, implementing task retries, handling task failures, integrating Apache Spark for data processing, and setting up sensors for monitoring and control. You will also engage in exercises like data validation and calculating average customer reviews.
Who is the target audience for this course?
This course is ideal for beginner data engineers who want to gain foundational knowledge in data orchestration. It is also suitable for experienced professionals looking to refine their skills in building scalable and reliable data processing systems using Apache Airflow.
How does the depth of this course compare to other data engineering courses?
The course offers a practical approach to mastering Apache Airflow, covering essential aspects like architecture, task retries, failure handling, and integration with Apache Spark. It provides a focused exploration of Airflow's capabilities, making it suitable for those specifically interested in orchestrating data workflows, rather than a broad survey of data engineering tools.
What specific tools or platforms will I learn to use in this course?
The course focuses on Apache Airflow and its integration with Apache Spark. You will learn to define and manage workflows using Airflow's DAGs, handle errors and retries, use sensors for monitoring, and automate data loading into data lakes. The course also touches on using Docker and Virtualenv as optional components.
What topics are not covered in this course?
The course does not cover in-depth data engineering topics outside of Apache Airflow. While it includes some interaction with Apache Spark and data ingestion techniques, it does not delve into other data processing frameworks or the broader ecosystem of data engineering tools. It focuses specifically on orchestrating workflows using Airflow.
How much time should I expect to commit to this course?
The course consists of 27 lessons. While the exact runtime is not specified, you should be prepared to dedicate a few hours each week to watch the lessons, complete the exercises, and implement the projects. The time commitment will vary depending on your prior experience and how deeply you engage with the material.