Azure Data Pipelines with Terraform

4h 20m 29s
English
Paid

Course description

Azure is becoming an increasingly popular platform for companies using the Microsoft365 ecosystem. If you want to enhance your data engineering skills, the ability to work with Azure and automate infrastructure using Terraform are key competencies. That is why we created this course "Azure ETL with Terraform".

In a practical project, you will learn how to build a comprehensive data processing solution in Azure, combining the capabilities of Terraform, Azure Data Factory, Synapse Analytics, and Power BI.

Read more about the course

You will create a fully automated ETL process:

  • Extract data from an external API
  • Process it using powerful Azure tools
  • Prepare the data for visualization

In the process, you will implement Lakehouse and Medallion architecture (Bronze, Silver, Gold layers) to make your pipeline efficient and scalable.

By the end of the course, you will not only master the principles of building modern data pipelines and infrastructure automation but also gain a comprehensive practical project for your portfolio.

What you will learn in the course

Introduction to Azure and Terraform

Get acquainted with Azure's role in the modern data landscape and key services for data engineers: Data Factory, Data Lake, and Synapse Analytics. Understand how Terraform helps manage infrastructure resources as code (IaC), making their creation and maintenance scalable and reliable.

Practical Setup

Install Terraform, configure it to work with Azure. Create a Service Principal, set up authentication for secure automated resource deployment, and prepare a working environment for resource management.

Basics of Terraform

Understand the structure of a Terraform project, learn the basic commands and principles of modular development.

Learn to:

  • Deploy Azure Data Factory for pipeline orchestration
  • Configure Azure Data Lake Storage for data storage (Bronze layer)
  • Deploy Synapse Analytics for data processing
  • Master writing reusable and scalable code in Terraform.

Real Deployment

Start deploying pipeline components: connect Azure Data Factory to an external Soccer API for data loading, configure Azure Data Lake for storing raw data. You will learn to combine manual and automated approaches as done in real projects.

CI/CD for Infrastructure

Understand how to apply CI/CD principles for infrastructure using Terraform and Azure DevOps. Learn:

  • Continuous Integration (CI): automatic build, testing, and code verification
  • Continuous Deployment (CD): automatic infrastructure deployment and application updates
  • Learn to integrate Terraform into CI/CD pipelines to ensure your deployments are stable, repeatable, and fast.

What’s Next

In the next parts of the course, you will dive deeper into:

  • API integration (using the Soccer API as an example)
  • Advanced features of Azure Data Factory for batch data processing
  • Advanced data processing in Synapse Spark
  • Optimizing Lakehouse architecture for handling large volumes of data and team collaboration
  • Full automation of deployment pipelines for replicating infrastructure across different environments

Watch Online

Join premium to watch
Go to premium
# Title Duration
1 Introduction 01:52
2 Software Setup 04:32
3 Introduction to Azure 01:44
4 Managing Azure 10:52
5 Introduction to Terraform 02:38
6 Terraform Setup on Azure 03:49
7 Terraform Project Structure 06:44
8 Terraform Commands 09:03
9 Backend Deployment 01:40
10 Terraform Modules 09:39
11 Service Principle Deployment 05:18
12 Why CI/CD 05:18
13 CI/CD Process Basics 04:55
14 CI/CD Steps 05:28
15 CI/CD Workflow Example 05:24
16 CI/CD Bascis Summary 01:23
17 Azure CI/CD Pipelines Terminology 10:22
18 Single YAML Pipeline Approach 07:31
19 Azure Dev Ops & Azure Cloud setup 08:27
20 CI/CD Pipeline Implementation 11:58
21 Pipeline Source Code explained & Job Analysis 14:08
22 Executing the CI/CD Pipeline 02:20
23 API Introduction 11:03
24 Azure Data Factory Introduction 05:39
25 Azure Data Factory Components 04:05
26 Working with Data Factory - 1 04:47
27 Working with Data Factory - 2 08:00
28 Working with Data Factory - 3 10:38
29 Working with Data Factory - 4 08:43
30 Introduction to Databricks 06:27
31 Databricks Infrastructure Setup - 1 11:03
32 Databricks Infrastructure Setup - 2 04:22
33 Databricks Infrastructure Setup - 3 04:52
34 The Databricks User Interface 08:10
35 End-To-End Pipeline Execution - 1 05:28
36 End-To-End Pipeline Execution - 2 04:34
37 End-To-End Pipeline Execution - 3 15:09
38 End-To-End Pipeline Execution - 4 06:41
39 End-To-End Pipeline Execution - 5 05:43

Comments

0 comments

Want to join the conversation?

Sign in to comment

Similar courses

Build Fast Masterclass

Build Fast Masterclass

Sources: BuildFast Academy
How to finally Launch your AI Product (without ripping your hair out).. that makes you money in 30 days (or less). But unlike other AI courses, you won't learn
7 hours 22 minutes 11 seconds
PyTorch for Deep Learning and Computer Vision

PyTorch for Deep Learning and Computer Vision

Sources: udemy
PyTorch has rapidly become one of the most transformative frameworks in the field of Deep Learning. Since its release, PyTorch has completely changed the landsc
10 hours 20 minutes 51 seconds
Building APIs with FastAPI

Building APIs with FastAPI

Sources: Andreas Kretz
API is the foundation of any modern data platform. You either provide an API for clients or use external APIs yourself. In any case, it's important to be...
1 hour 35 minutes 40 seconds
Python for Data Science and Machine Learning Bootcamp

Python for Data Science and Machine Learning Bootcamp

Sources: udemy
Are you ready to start your path to becoming a Data Scientist! This comprehensive course will be your guide to learning how to use the power of Python to analy
24 hours 49 minutes 42 seconds
Case Study in A/B Testing

Case Study in A/B Testing

Sources: LunarTech
Examples from practice in A/B testing - this course will introduce you to the methods of designing, conducting, and analyzing experiments using A/B...
1 hour 56 minutes 17 seconds