Apache Airflow Workflow Orchestration
Apache Airflow is a platform-independent tool for workflow orchestration that provides extensive capabilities for creating and monitoring both streaming and batch pipelines. Even the most complex processes are easily implemented with its help—all with the support of key platforms and tools in the world of Data Engineering, including AWS, Google Cloud, and others.
Airflow not only allows for scheduling and managing processes but also tracking job execution in real-time, as well as quickly identifying and resolving errors.
In brief: today, Airflow is one of the most in-demand and "hyped" tools in the field of pipeline orchestration. It is actively used by companies worldwide, and knowledge of Airflow is becoming an important skill for any data engineer. This is especially relevant for students starting their journey in this field.
Read more about the course
Basic Concepts of Airflow
Introduction to the fundamentals of working with Airflow: you will learn how DAGs (Directed Acyclic Graphs) are created, what they consist of (operators, tasks), and how the architecture of Airflow is structured - including the database, scheduler, and web interface. We will also look at examples of event-driven pipelines that can be implemented using Airflow.
Installation and Environment Setup
In practice, you will work on a project dealing with weather data processing. The DAG will fetch data from a weather API, transform it, and store it in a Postgres database. You will learn how to:
- configure the environment using Docker;
- verify the web interface and container operations;
- configure the API and create the necessary tables in the database.
Practice: Creating DAGs
You will thoroughly understand the Airflow interface and learn to monitor task statuses. Then you will:
- create DAGs based on Airflow 2.0 that retrieve and process data;
- master the Taskflow API - a modern approach to building DAGs with more convenient syntax;
- implement parallel task execution (fanout) to run multiple processes simultaneously.
Watch Online Apache Airflow Workflow Orchestration
| # | Title | Duration |
|---|---|---|
| 1 | Introduction | 01:37 |
| 2 | Airflow Usage | 03:20 |
| 3 | Fundamental Concepts | 02:48 |
| 4 | Airflow Architecture | 03:10 |
| 5 | Example Pipelines | 04:50 |
| 6 | Spotlight 3rd Party Operators | 02:18 |
| 7 | Airflow XComs | 04:33 |
| 8 | Project Setup | 01:44 |
| 9 | Docker Setup Explained | 02:07 |
| 10 | Docker Compose & Starting Containers | 04:24 |
| 11 | Checking Services | 01:49 |
| 12 | Setup WeatherAPI | 01:34 |
| 13 | Setup Postgres DB | 01:59 |
| 14 | Airflow Webinterface | 04:38 |
| 15 | Creating DAG With Airflow 2.0 | 09:47 |
| 16 | Running our DAG | 04:16 |
| 17 | Creating DAG With TaskflowAPI | 07:00 |
| 18 | Getting Data From the API With SimpleHTTPOperator | 03:39 |
| 19 | Writing into Postgres | 04:13 |
| 20 | Parallel Processing | 04:16 |
| 21 | Recap & Outlook | 04:39 |
Similar courses to Apache Airflow Workflow Orchestration
Business Intelligence with Excelzerotomastery.io
Case Study in Product Data ScienceLunarTech
Platform & Pipeline SecurityAndreas Kretz
Relational Data ModelingEka Ponkratova
Spark and Python for Big Data with PySparkudemy
Introduction to Data Engineeringzerotomastery.io
Streaming with Kafka & SparkAndreas Kretz
MongoDB FundamentalsAndreas Kretz
