Data Engineering on Azure

1h 20m 57s
English
Paid

Microsoft Azure is a versatile cloud platform offering over 200 products and services specifically designed for data storage, management, virtual machine deployment, and application development in the cloud. Azure's flexibility supports various frameworks and tools, enabling applications to run seamlessly in a multi-cloud environment, locally, or at the network edge.

Course Overview

Join Kristian Bakarich in this hands-on course to create a robust streaming data processing pipeline in Azure. You'll dive into using key Azure services to process Twitter data streams formatted in JSON, enhancing your expertise in cloud solutions.

Key Azure Services You Will Use

  • APIM (API Management) - Manage data intake efficiently.
  • Blob Storage - Secure and scalable storage solution.
  • Azure Functions - Execute processing logic seamlessly.
  • Cosmos DB - Store processed data effectively.
  • Power BI - Visualize data insights dynamically.

Course Structure

1. Introduction and Architecture

Become familiar with the complete solution architecture and understand the primary components of the data pipeline.

2. Data Creation and Sending

Create a JSON file with messages and develop a Python script to send these JSON objects via HTTP requests to Azure API Management.

3. Development and Deployment of Azure Functions

Learn how to create and deploy Azure functions in Python using Visual Studio Code, constructing a function project with essential logic.

4. Service Integration

Integrate Event Hubs, Azure Functions, and Cosmos DB by learning to manage message flow from the Event Hub to Cosmos DB.

5. Data Visualization with Power BI

Connect Power BI Desktop to Cosmos DB for real-time data visualization and analytics.

Prerequisites

Before embarking on this course, ensure you have the following:

  • An active Azure account
  • Basic programming skills, particularly in Python
  • Foundational knowledge of data storage solutions
  • Understanding of API concepts (consider our recommended course: "Designing and Developing APIs with FastAPI")
  • Basic familiarity with message queues

About the Author: Kristijan Bakarić

Kristijan Bakarić thumbnail

Kristian is a specialist combining scientific and engineering approaches, working at an international energy company since 2013.

Half of his career was dedicated to oil and gas exploration as a geologist-geophysicist, and the other half to analytics and engineering projects in subsurface data processing. In recent years, he has been actively involved in the digitalization and automation of operations.

Kristian possesses extensive knowledge in various fields, tools, and technologies. His current focus is on data engineering and cloud solutions on the Azure platform.

He is passionate about data, technology, and learning, and enjoys sharing his knowledge and experience with others.

Watch Online 14 lessons

This is a demo lesson (10:00 remaining)

You can watch up to 10 minutes for free. Subscribe to unlock all 14 lessons in this course and access 10,000+ hours of premium content across all courses.

View Pricing
0:00
/
#1: Data Engineering in Azure - Streaming Data Pipelines
All Course Lessons (14)
#Lesson TitleDurationAccess
1
Data Engineering in Azure - Streaming Data Pipelines Demo
02:44
2
Introduction to Datasets and Local Preprocessing
07:07
3
Deploying your Code on Visual Studio to Docker containers
05:28
4
Develop Azure Functions via Python and VS Code
05:53
5
Deploy Azure Function to Azure Function App and Test it
06:27
6
Integrate Azure Function with Blob Storage via bindings
04:59
7
Expose Azure Function as a Backend, and Test it from Insomnia
07:06
8
Securely Store Secrets in Azure Key Vault and Connect APIM to Key Vault
04:42
9
Add Basic authentication in API Management using Key Vault and Named Values
04:36
10
Test APIM and Imported Azure Function App and Function via Local Python Program
02:35
11
Create Event Hubs and Test Capture Events Feature
07:00
12
Modify Existing Azure Function to Include Event Hubs Binding and Test It
06:43
13
Create a Cosmos DB (Core SQL) and Create a New Azure Function that writes Messages to Cosmos DB
09:04
14
Connect Power Bi Desktop via Connector, and create a dashboard
06:33
Unlock unlimited learning

Get instant access to all 13 lessons in this course, plus thousands of other premium courses. One subscription, unlimited knowledge.

Learn more about subscription