Azure Data Pipelines with Terraform

4h 20m 29s
English
Paid

Course description

Azure is becoming an increasingly popular platform for companies using the Microsoft365 ecosystem. If you want to enhance your data engineering skills, the ability to work with Azure and automate infrastructure using Terraform are key competencies. That is why we created this course "Azure ETL with Terraform".

In a practical project, you will learn how to build a comprehensive data processing solution in Azure, combining the capabilities of Terraform, Azure Data Factory, Synapse Analytics, and Power BI.

Read more about the course

You will create a fully automated ETL process:

  • Extract data from an external API
  • Process it using powerful Azure tools
  • Prepare the data for visualization

In the process, you will implement Lakehouse and Medallion architecture (Bronze, Silver, Gold layers) to make your pipeline efficient and scalable.

By the end of the course, you will not only master the principles of building modern data pipelines and infrastructure automation but also gain a comprehensive practical project for your portfolio.

What you will learn in the course

Introduction to Azure and Terraform

Get acquainted with Azure's role in the modern data landscape and key services for data engineers: Data Factory, Data Lake, and Synapse Analytics. Understand how Terraform helps manage infrastructure resources as code (IaC), making their creation and maintenance scalable and reliable.

Practical Setup

Install Terraform, configure it to work with Azure. Create a Service Principal, set up authentication for secure automated resource deployment, and prepare a working environment for resource management.

Basics of Terraform

Understand the structure of a Terraform project, learn the basic commands and principles of modular development.

Learn to:

  • Deploy Azure Data Factory for pipeline orchestration
  • Configure Azure Data Lake Storage for data storage (Bronze layer)
  • Deploy Synapse Analytics for data processing
  • Master writing reusable and scalable code in Terraform.

Real Deployment

Start deploying pipeline components: connect Azure Data Factory to an external Soccer API for data loading, configure Azure Data Lake for storing raw data. You will learn to combine manual and automated approaches as done in real projects.

CI/CD for Infrastructure

Understand how to apply CI/CD principles for infrastructure using Terraform and Azure DevOps. Learn:

  • Continuous Integration (CI): automatic build, testing, and code verification
  • Continuous Deployment (CD): automatic infrastructure deployment and application updates
  • Learn to integrate Terraform into CI/CD pipelines to ensure your deployments are stable, repeatable, and fast.

What’s Next

In the next parts of the course, you will dive deeper into:

  • API integration (using the Soccer API as an example)
  • Advanced features of Azure Data Factory for batch data processing
  • Advanced data processing in Synapse Spark
  • Optimizing Lakehouse architecture for handling large volumes of data and team collaboration
  • Full automation of deployment pipelines for replicating infrastructure across different environments

Watch Online

This is a demo lesson (10:00 remaining)

You can watch up to 10 minutes for free. Subscribe to unlock all 39 lessons in this course and access 10,000+ hours of premium content across all courses.

View Pricing

Watch Online Azure Data Pipelines with Terraform

0:00
/
#1: Introduction

All Course Lessons (39)

#Lesson TitleDurationAccess
1
Introduction Demo
01:52
2
Software Setup
04:32
3
Introduction to Azure
01:44
4
Managing Azure
10:52
5
Introduction to Terraform
02:38
6
Terraform Setup on Azure
03:49
7
Terraform Project Structure
06:44
8
Terraform Commands
09:03
9
Backend Deployment
01:40
10
Terraform Modules
09:39
11
Service Principle Deployment
05:18
12
Why CI/CD
05:18
13
CI/CD Process Basics
04:55
14
CI/CD Steps
05:28
15
CI/CD Workflow Example
05:24
16
CI/CD Bascis Summary
01:23
17
Azure CI/CD Pipelines Terminology
10:22
18
Single YAML Pipeline Approach
07:31
19
Azure Dev Ops & Azure Cloud setup
08:27
20
CI/CD Pipeline Implementation
11:58
21
Pipeline Source Code explained & Job Analysis
14:08
22
Executing the CI/CD Pipeline
02:20
23
API Introduction
11:03
24
Azure Data Factory Introduction
05:39
25
Azure Data Factory Components
04:05
26
Working with Data Factory - 1
04:47
27
Working with Data Factory - 2
08:00
28
Working with Data Factory - 3
10:38
29
Working with Data Factory - 4
08:43
30
Introduction to Databricks
06:27
31
Databricks Infrastructure Setup - 1
11:03
32
Databricks Infrastructure Setup - 2
04:22
33
Databricks Infrastructure Setup - 3
04:52
34
The Databricks User Interface
08:10
35
End-To-End Pipeline Execution - 1
05:28
36
End-To-End Pipeline Execution - 2
04:34
37
End-To-End Pipeline Execution - 3
15:09
38
End-To-End Pipeline Execution - 4
06:41
39
End-To-End Pipeline Execution - 5
05:43

Unlock unlimited learning

Get instant access to all 38 lessons in this course, plus thousands of other premium courses. One subscription, unlimited knowledge.

Learn more about subscription

Comments

0 comments

Want to join the conversation?

Sign in to comment

Similar courses

Getting Started with Embedded AI | Edge AI

Getting Started with Embedded AI | Edge AI

Sources: udemy
Nowadays, you may have heard of many keywords like Embedded AI /Embedded ML /Edge AI, the meaning behind them is the same, I.e. To make an AI algorithm or model
3 hours 33 minutes 42 seconds
Data Engineering on GCP

Data Engineering on GCP

Sources: Andreas Kretz
Google Cloud Platform (GCP) is one of the most popular cloud platforms in the world, providing an extensive set of tools and services for building...
1 hour 17 minutes 33 seconds
Complete Terraform Course - Beginner to Advanced

Complete Terraform Course - Beginner to Advanced

Sources: udemy, Techworld with Nana
Learn Terraform with hands-on demos by automating AWS infrastructure. You will learn all the concepts with practical examples by automating the AWS infrastructu
6 hours 24 minutes 44 seconds
Machine Learning A-Z : Become Kaggle Master

Machine Learning A-Z : Become Kaggle Master

Sources: udemy
Want to become a good Data Scientist? Then this is a right course for you. This course has been designed by IIT professionals who have mastered in Mathematics and Data Science....
36 hours 23 minutes 54 seconds
Python for Data Science and Machine Learning Bootcamp

Python for Data Science and Machine Learning Bootcamp

Sources: udemy
Are you ready to start your path to becoming a Data Scientist! This comprehensive course will be your guide to learning how to use the power of Python to analy
24 hours 49 minutes 42 seconds