CF

Data Engineering on Azure

1h 20m 57s
English
Paid

Microsoft Azure is a versatile cloud platform offering over 200 products and services specifically designed for data storage, management, virtual machine deployment, and application development in the cloud. Azure's flexibility supports various frameworks and tools, enabling applications to run seamlessly in a multi-cloud environment, locally, or at the network edge.

Course Overview

Join Kristian Bakarich in this hands-on course to create a robust streaming data processing pipeline in Azure. You'll dive into using key Azure services to process Twitter data streams formatted in JSON, enhancing your expertise in cloud solutions.

Key Azure Services You Will Use

  • APIM (API Management) - Manage data intake efficiently.
  • Blob Storage - Secure and scalable storage solution.
  • Azure Functions - Execute processing logic seamlessly.
  • Cosmos DB - Store processed data effectively.
  • Power BI - Visualize data insights dynamically.

Course Structure

1. Introduction and Architecture

Become familiar with the complete solution architecture and understand the primary components of the data pipeline.

2. Data Creation and Sending

Create a JSON file with messages and develop a Python script to send these JSON objects via HTTP requests to Azure API Management.

3. Development and Deployment of Azure Functions

Learn how to create and deploy Azure functions in Python using Visual Studio Code, constructing a function project with essential logic.

4. Service Integration

Integrate Event Hubs, Azure Functions, and Cosmos DB by learning to manage message flow from the Event Hub to Cosmos DB.

5. Data Visualization with Power BI

Connect Power BI Desktop to Cosmos DB for real-time data visualization and analytics.

Prerequisites

Before embarking on this course, ensure you have the following:

  • An active Azure account
  • Basic programming skills, particularly in Python
  • Foundational knowledge of data storage solutions
  • Understanding of API concepts (consider our recommended course: "Designing and Developing APIs with FastAPI")
  • Basic familiarity with message queues

Additional

https://github.com/team-data-science/azure-data-engineering

About the Author: Kristijan Bakarić

Kristijan Bakarić thumbnail

Kristijan Bakarić is a Croatian data engineer and educator focused on the Microsoft Azure data-engineering stack — the combination of Azure-specific data services (Data Factory, Synapse, Databricks on Azure) that anchors enterprise data work on the Microsoft cloud.

His CourseFlix listing carries Data Engineering on Azure — a structured treatment of building production data pipelines on Azure: the service selection trade-offs, the orchestration patterns, the storage layers (ADLS, Delta), and the operational patterns for running Azure data infrastructure.

Material is paid and aimed at data engineers working on the Microsoft Azure cloud. For broader content, see CourseFlix's Azure and Data processing and analysis category pages.

Watch Online 14 lessons

This is a demo lesson (10:00 remaining)

You can watch up to 10 minutes for free. Subscribe to unlock all 14 lessons in this course and access 10,000+ hours of premium content across all courses.

View Pricing
0:00
/
#1: Data Engineering in Azure - Streaming Data Pipelines
All Course Lessons (14)
#Lesson TitleDurationAccess
1
Data Engineering in Azure - Streaming Data Pipelines Demo
02:44
2
Introduction to Datasets and Local Preprocessing
07:07
3
Deploying your Code on Visual Studio to Docker containers
05:28
4
Develop Azure Functions via Python and VS Code
05:53
5
Deploy Azure Function to Azure Function App and Test it
06:27
6
Integrate Azure Function with Blob Storage via bindings
04:59
7
Expose Azure Function as a Backend, and Test it from Insomnia
07:06
8
Securely Store Secrets in Azure Key Vault and Connect APIM to Key Vault
04:42
9
Add Basic authentication in API Management using Key Vault and Named Values
04:36
10
Test APIM and Imported Azure Function App and Function via Local Python Program
02:35
11
Create Event Hubs and Test Capture Events Feature
07:00
12
Modify Existing Azure Function to Include Event Hubs Binding and Test It
06:43
13
Create a Cosmos DB (Core SQL) and Create a New Azure Function that writes Messages to Cosmos DB
09:04
14
Connect Power Bi Desktop via Connector, and create a dashboard
06:33
Unlock unlimited learning

Get instant access to all 13 lessons in this course, plus thousands of other premium courses. One subscription, unlimited knowledge.

Learn more about subscription

Related courses

Frequently asked questions

What are the prerequisites for enrolling in this course?
To enroll in this course, you should have a basic understanding of cloud platforms and some experience with Python programming. Familiarity with Microsoft Azure services is beneficial but not required, as the course will guide you through using Azure-specific tools such as Azure Functions and Cosmos DB.
What kind of projects will I build during the course?
During the course, you will build a streaming data processing pipeline using Azure services. This involves creating a Python script to send JSON data streams to Azure API Management, deploying Azure Functions, integrating with Blob Storage, and visualizing data with Power BI. A key project is developing a function that processes Twitter data streams and stores them in Cosmos DB.
Who is the target audience for this course?
This course is designed for data engineers and developers interested in building scalable data solutions using Microsoft Azure. It is also suitable for IT professionals who wish to enhance their skills in cloud-based data processing and visualization.
How does the course's depth compare to similar courses?
The course offers a detailed exploration of building a data processing pipeline on Azure, focusing on hands-on implementation using Azure Functions, Blob Storage, and Cosmos DB. Unlike general cloud courses, it provides specific insights into integrating Azure services for real-time data processing, making it more specialized in streaming data solutions.
What tools and platforms will I learn to use?
You will learn to use several key Azure services including API Management for data intake, Blob Storage for data storage, Azure Functions for executing processing logic, Cosmos DB for data storage, and Power BI for data visualization. Additionally, tools like Visual Studio Code and Docker containers are part of the development process.
What topics are not covered in this course?
The course does not cover topics outside the scope of Microsoft Azure, such as other cloud providers like AWS or Google Cloud. It also does not cover advanced data science techniques or in-depth machine learning algorithms, focusing instead on building and integrating Azure-based data pipelines.
What is the expected time commitment for the course?
The course consists of 14 lessons and, while the exact runtime is unspecified, you can expect to spend several hours on each lesson. This includes time for lectures, hands-on projects, and practical exercises to apply what you have learned.