Azure Data Pipelines with Terraform

2h 15m 58s
English
Paid

Azure is becoming an increasingly popular platform for companies using the Microsoft365 ecosystem. If you want to enhance your data engineering skills, the ability to work with Azure and automate infrastructure using Terraform are key competencies. That is why we created this course "Azure ETL with Terraform".

In a practical project, you will learn how to build a comprehensive data processing solution in Azure, combining the capabilities of Terraform, Azure Data Factory, Synapse Analytics, and Power BI.

Read more about the course

You will create a fully automated ETL process:

  • Extract data from an external API
  • Process it using powerful Azure tools
  • Prepare the data for visualization

In the process, you will implement Lakehouse and Medallion architecture (Bronze, Silver, Gold layers) to make your pipeline efficient and scalable.

By the end of the course, you will not only master the principles of building modern data pipelines and infrastructure automation but also gain a comprehensive practical project for your portfolio.

What you will learn in the course

Introduction to Azure and Terraform

Get acquainted with Azure's role in the modern data landscape and key services for data engineers: Data Factory, Data Lake, and Synapse Analytics. Understand how Terraform helps manage infrastructure resources as code (IaC), making their creation and maintenance scalable and reliable.

Practical Setup

Install Terraform, configure it to work with Azure. Create a Service Principal, set up authentication for secure automated resource deployment, and prepare a working environment for resource management.

Basics of Terraform

Understand the structure of a Terraform project, learn the basic commands and principles of modular development.

Learn to:

  • Deploy Azure Data Factory for pipeline orchestration
  • Configure Azure Data Lake Storage for data storage (Bronze layer)
  • Deploy Synapse Analytics for data processing
  • Master writing reusable and scalable code in Terraform.

Real Deployment

Start deploying pipeline components: connect Azure Data Factory to an external Soccer API for data loading, configure Azure Data Lake for storing raw data. You will learn to combine manual and automated approaches as done in real projects.

CI/CD for Infrastructure

Understand how to apply CI/CD principles for infrastructure using Terraform and Azure DevOps. Learn:

  • Continuous Integration (CI): automatic build, testing, and code verification
  • Continuous Deployment (CD): automatic infrastructure deployment and application updates
  • Learn to integrate Terraform into CI/CD pipelines to ensure your deployments are stable, repeatable, and fast.

What’s Next

In the next parts of the course, you will dive deeper into:

  • API integration (using the Soccer API as an example)
  • Advanced features of Azure Data Factory for batch data processing
  • Advanced data processing in Synapse Spark
  • Optimizing Lakehouse architecture for handling large volumes of data and team collaboration
  • Full automation of deployment pipelines for replicating infrastructure across different environments

Watch Online Azure Data Pipelines with Terraform

Join premium to watch
Go to premium
# Title Duration
1 Introduction 01:52
2 Software Setup 04:32
3 Introduction to Azure 01:44
4 Managing Azure 10:52
5 Introduction to Terraform 02:38
6 Terraform Setup on Azure 03:49
7 Terraform Project Structure 06:44
8 Terraform Commands 09:03
9 Backend Deployment 01:40
10 Terraform Modules 09:39
11 Service Principle Deployment 05:18
12 Why CI/CD 05:18
13 CI/CD Process Basics 04:55
14 CI/CD Steps 05:28
15 CI/CD Workflow Example 05:24
16 CI/CD Bascis Summary 01:23
17 Azure CI/CD Pipelines Terminology 10:22
18 Single YAML Pipeline Approach 07:31
19 Azure Dev Ops & Azure Cloud setup 08:27
20 CI/CD Pipeline Implementation 11:58
21 Pipeline Source Code explained & Job Analysis 14:08
22 Executing the CI/CD Pipeline 02:20
23 Preview of what's coming next in April 00:53

Similar courses to Azure Data Pipelines with Terraform

Machine Learning with Python : COMPLETE COURSE FOR BEGINNERS

Machine Learning with Python : COMPLETE COURSE FOR BEGINNERSudemy

Category: Python, Data processing and analysis
Duration 13 hours 12 minutes 31 seconds
Apache Airflow Workflow Orchestration

Apache Airflow Workflow OrchestrationAndreas Kretz

Category: Other (Tools), Data processing and analysis
Duration 1 hour 18 minutes 41 seconds
Learning Apache Spark

Learning Apache SparkAndreas Kretz

Category: Data processing and analysis
Duration 1 hour 44 minutes 4 seconds
Azure Data Engineer Workshop In A Weekend

Azure Data Engineer Workshop In A Weekendudemy

Category: Azure
Duration 9 hours 18 minutes 40 seconds
Streaming with Kafka & Spark

Streaming with Kafka & SparkAndreas Kretz

Category: Data processing and analysis
Duration 2 hours 46 minutes 25 seconds
Microsoft Azure Administrator (AZ-104)

Microsoft Azure Administrator (AZ-104)James Lee

Category: Azure
Duration 28 hours 35 minutes 23 seconds
TensorFlow Developer Certificate in 2023: Zero to Mastery

TensorFlow Developer Certificate in 2023: Zero to Masteryzerotomastery.io

Category: Data processing and analysis
Duration 62 hours 43 minutes 54 seconds
Case Study in Product Data Science

Case Study in Product Data ScienceLunarTech

Category: Data processing and analysis
Duration 1 hour 4 minutes 47 seconds
Introduction to Data Engineering 2025

Introduction to Data Engineering 2025Andreas Kretz

Category: Data processing and analysis
Duration 44 minutes 26 seconds
The Data Bootcamp: Transform your Data using dbt™

The Data Bootcamp: Transform your Data using dbt™udemy

Category: Data processing and analysis
Duration 4 hours 10 minutes 51 seconds