Azure Data Pipelines with Terraform

You will create a fully automated ETL process:

• Extract data from an external API
• Process it using powerful Azure tools
• Prepare the data for visualization

In the process, you will implement Lakehouse and Medallion architecture (Bronze, Silver, Gold layers) to make your pipeline efficient and scalable.

By the end of the course, you will not only master the principles of building modern data pipelines and infrastructure automation but also gain a comprehensive practical project for your portfolio.

What you will learn in the course

Introduction to Azure and Terraform

Get acquainted with Azure's role in the modern data landscape and key services for data engineers: Data Factory, Data Lake, and Synapse Analytics. Understand how Terraform helps manage infrastructure resources as code (IaC), making their creation and maintenance scalable and reliable.

Practical Setup

Install Terraform, configure it to work with Azure. Create a Service Principal, set up authentication for secure automated resource deployment, and prepare a working environment for resource management.

Basics of Terraform

Understand the structure of a Terraform project, learn the basic commands and principles of modular development.

Learn to:

• Deploy Azure Data Factory for pipeline orchestration
• Configure Azure Data Lake Storage for data storage (Bronze layer)
• Deploy Synapse Analytics for data processing
• Master writing reusable and scalable code in Terraform.

Real Deployment

Start deploying pipeline components: connect Azure Data Factory to an external Soccer API for data loading, configure Azure Data Lake for storing raw data. You will learn to combine manual and automated approaches as done in real projects.

CI/CD for Infrastructure

Understand how to apply CI/CD principles for infrastructure using Terraform and Azure DevOps. Learn:

• Continuous Integration (CI): automatic build, testing, and code verification
• Continuous Deployment (CD): automatic infrastructure deployment and application updates
• Learn to integrate Terraform into CI/CD pipelines to ensure your deployments are stable, repeatable, and fast.

What’s Next

In the next parts of the course, you will dive deeper into:

• API integration (using the Soccer API as an example)
• Advanced features of Azure Data Factory for batch data processing
• Advanced data processing in Synapse Spark
• Optimizing Lakehouse architecture for handling large volumes of data and team collaboration
• Full automation of deployment pipelines for replicating infrastructure across different environments