Azure Data Pipelines with Terraform
Course description
Azure is becoming an increasingly popular platform for companies using the Microsoft365 ecosystem. If you want to enhance your data engineering skills, the ability to work with Azure and automate infrastructure using Terraform are key competencies. That is why we created this course "Azure ETL with Terraform".
In a practical project, you will learn how to build a comprehensive data processing solution in Azure, combining the capabilities of Terraform, Azure Data Factory, Synapse Analytics, and Power BI.
Read more about the course
You will create a fully automated ETL process:
- Extract data from an external API
- Process it using powerful Azure tools
- Prepare the data for visualization
In the process, you will implement Lakehouse and Medallion architecture (Bronze, Silver, Gold layers) to make your pipeline efficient and scalable.
By the end of the course, you will not only master the principles of building modern data pipelines and infrastructure automation but also gain a comprehensive practical project for your portfolio.
What you will learn in the course
Introduction to Azure and Terraform
Get acquainted with Azure's role in the modern data landscape and key services for data engineers: Data Factory, Data Lake, and Synapse Analytics. Understand how Terraform helps manage infrastructure resources as code (IaC), making their creation and maintenance scalable and reliable.
Practical Setup
Install Terraform, configure it to work with Azure. Create a Service Principal, set up authentication for secure automated resource deployment, and prepare a working environment for resource management.
Basics of Terraform
Understand the structure of a Terraform project, learn the basic commands and principles of modular development.
Learn to:
- Deploy Azure Data Factory for pipeline orchestration
- Configure Azure Data Lake Storage for data storage (Bronze layer)
- Deploy Synapse Analytics for data processing
- Master writing reusable and scalable code in Terraform.
Real Deployment
Start deploying pipeline components: connect Azure Data Factory to an external Soccer API for data loading, configure Azure Data Lake for storing raw data. You will learn to combine manual and automated approaches as done in real projects.
CI/CD for Infrastructure
Understand how to apply CI/CD principles for infrastructure using Terraform and Azure DevOps. Learn:
- Continuous Integration (CI): automatic build, testing, and code verification
- Continuous Deployment (CD): automatic infrastructure deployment and application updates
- Learn to integrate Terraform into CI/CD pipelines to ensure your deployments are stable, repeatable, and fast.
What’s Next
In the next parts of the course, you will dive deeper into:
- API integration (using the Soccer API as an example)
- Advanced features of Azure Data Factory for batch data processing
- Advanced data processing in Synapse Spark
- Optimizing Lakehouse architecture for handling large volumes of data and team collaboration
- Full automation of deployment pipelines for replicating infrastructure across different environments
Watch Online
All Course Lessons (39)
| # | Lesson Title | Duration | Access |
|---|---|---|---|
| 1 | Introduction Demo | 01:52 | |
| 2 | Software Setup | 04:32 | |
| 3 | Introduction to Azure | 01:44 | |
| 4 | Managing Azure | 10:52 | |
| 5 | Introduction to Terraform | 02:38 | |
| 6 | Terraform Setup on Azure | 03:49 | |
| 7 | Terraform Project Structure | 06:44 | |
| 8 | Terraform Commands | 09:03 | |
| 9 | Backend Deployment | 01:40 | |
| 10 | Terraform Modules | 09:39 | |
| 11 | Service Principle Deployment | 05:18 | |
| 12 | Why CI/CD | 05:18 | |
| 13 | CI/CD Process Basics | 04:55 | |
| 14 | CI/CD Steps | 05:28 | |
| 15 | CI/CD Workflow Example | 05:24 | |
| 16 | CI/CD Bascis Summary | 01:23 | |
| 17 | Azure CI/CD Pipelines Terminology | 10:22 | |
| 18 | Single YAML Pipeline Approach | 07:31 | |
| 19 | Azure Dev Ops & Azure Cloud setup | 08:27 | |
| 20 | CI/CD Pipeline Implementation | 11:58 | |
| 21 | Pipeline Source Code explained & Job Analysis | 14:08 | |
| 22 | Executing the CI/CD Pipeline | 02:20 | |
| 23 | API Introduction | 11:03 | |
| 24 | Azure Data Factory Introduction | 05:39 | |
| 25 | Azure Data Factory Components | 04:05 | |
| 26 | Working with Data Factory - 1 | 04:47 | |
| 27 | Working with Data Factory - 2 | 08:00 | |
| 28 | Working with Data Factory - 3 | 10:38 | |
| 29 | Working with Data Factory - 4 | 08:43 | |
| 30 | Introduction to Databricks | 06:27 | |
| 31 | Databricks Infrastructure Setup - 1 | 11:03 | |
| 32 | Databricks Infrastructure Setup - 2 | 04:22 | |
| 33 | Databricks Infrastructure Setup - 3 | 04:52 | |
| 34 | The Databricks User Interface | 08:10 | |
| 35 | End-To-End Pipeline Execution - 1 | 05:28 | |
| 36 | End-To-End Pipeline Execution - 2 | 04:34 | |
| 37 | End-To-End Pipeline Execution - 3 | 15:09 | |
| 38 | End-To-End Pipeline Execution - 4 | 06:41 | |
| 39 | End-To-End Pipeline Execution - 5 | 05:43 |
Unlock unlimited learning
Get instant access to all 38 lessons in this course, plus thousands of other premium courses. One subscription, unlimited knowledge.
Learn more about subscriptionComments
0 commentsWant to join the conversation?
Sign in to commentSimilar courses
The Data Bootcamp: Transform your Data using dbt™
Data Engineering on Azure
Microsoft DevOps Engineer (AZ-400)
Snowflake for Data Engineers
