Apache Airflow Workflow Orchestration

1h 18m 41s
English
Paid

Apache Airflow is a platform-independent tool for workflow orchestration that provides extensive capabilities for creating and monitoring both streaming and batch pipelines. Even the most complex processes are easily implemented with its help—all with the support of key platforms and tools in the world of Data Engineering, including AWS, Google Cloud, and others.

Airflow not only allows for scheduling and managing processes but also tracking job execution in real-time, as well as quickly identifying and resolving errors.

In brief: today, Airflow is one of the most in-demand and "hyped" tools in the field of pipeline orchestration. It is actively used by companies worldwide, and knowledge of Airflow is becoming an important skill for any data engineer. This is especially relevant for students starting their journey in this field.

Read more about the course

Basic Concepts of Airflow

Introduction to the fundamentals of working with Airflow: you will learn how DAGs (Directed Acyclic Graphs) are created, what they consist of (operators, tasks), and how the architecture of Airflow is structured - including the database, scheduler, and web interface. We will also look at examples of event-driven pipelines that can be implemented using Airflow.

Installation and Environment Setup

In practice, you will work on a project dealing with weather data processing. The DAG will fetch data from a weather API, transform it, and store it in a Postgres database. You will learn how to:

  • configure the environment using Docker;
  • verify the web interface and container operations;
  • configure the API and create the necessary tables in the database.

Practice: Creating DAGs

You will thoroughly understand the Airflow interface and learn to monitor task statuses. Then you will:

  • create DAGs based on Airflow 2.0 that retrieve and process data;
  • master the Taskflow API - a modern approach to building DAGs with more convenient syntax;
  • implement parallel task execution (fanout) to run multiple processes simultaneously.

Watch Online Apache Airflow Workflow Orchestration

Join premium to watch
Go to premium
# Title Duration
1 Introduction 01:37
2 Airflow Usage 03:20
3 Fundamental Concepts 02:48
4 Airflow Architecture 03:10
5 Example Pipelines 04:50
6 Spotlight 3rd Party Operators 02:18
7 Airflow XComs 04:33
8 Project Setup 01:44
9 Docker Setup Explained 02:07
10 Docker Compose & Starting Containers 04:24
11 Checking Services 01:49
12 Setup WeatherAPI 01:34
13 Setup Postgres DB 01:59
14 Airflow Webinterface 04:38
15 Creating DAG With Airflow 2.0 09:47
16 Running our DAG 04:16
17 Creating DAG With TaskflowAPI 07:00
18 Getting Data From the API With SimpleHTTPOperator 03:39
19 Writing into Postgres 04:13
20 Parallel Processing 04:16
21 Recap & Outlook 04:39

Similar courses to Apache Airflow Workflow Orchestration

Data Analysis for Beginners: Excel & Pivot Tables

Data Analysis for Beginners: Excel & Pivot Tableszerotomastery.io

Category: Data processing and analysis
Duration 2 hours 10 minutes 21 seconds
The Data Science Course: Complete Data Science Bootcamp 2023

The Data Science Course: Complete Data Science Bootcamp 2023udemy

Category: Data processing and analysis
Duration 31 hours 14 minutes 14 seconds
Platform & Pipeline Security

Platform & Pipeline SecurityAndreas Kretz

Category: Data processing and analysis
Duration 34 minutes 46 seconds
MobX In Depth With React(Hooks+TypeScript)

MobX In Depth With React(Hooks+TypeScript)udemy

Category: React.js, Other (Tools)
Duration 2 hours 25 minutes 54 seconds
Data Engineering on GCP

Data Engineering on GCPAndreas Kretz

Category: Data processing and analysis
Duration 1 hour 17 minutes 33 seconds
Istio Hands-On for Kubernetes

Istio Hands-On for Kubernetesudemy

Category: Kubernetes, Other (Tools)
Duration 12 hours 15 minutes 1 second
Case Study in Causal Analysis

Case Study in Causal AnalysisLunarTech

Category: Data processing and analysis
Duration 2 hours 3 minutes 34 seconds
Data Analysis with Pandas and Python

Data Analysis with Pandas and Pythonudemy

Category: Python, Data processing and analysis
Duration 19 hours 5 minutes 40 seconds
Introduction to Excel Automation: Excel Macros and VBA

Introduction to Excel Automation: Excel Macros and VBAzerotomastery.io

Category: Other (Tools)
Duration 2 hours 44 minutes 10 seconds