Data Engineering on Azure

1h 20m 57s
English
Paid

Course description

Microsoft Azure is a cloud platform offering over 200 products and services for data storage, management, virtual machine deployment, and application development in the cloud. Azure supports working with various frameworks and tools, allowing applications to run in a multi-cloud environment, locally, or at the network edge.

Read more about the course

What you will learn in the course

In this course guided by Kristian Bakarich, you will practically create a streaming data processing pipeline in Azure. As part of the project, you will learn to use key Azure services for processing Twitter data streams in JSON format, including:

  • APIM (API Management) - for data intake,
  • Blob Storage - for storage,
  • Azure Functions - for processing,
  • Cosmos DB - for storing processed data,
  • Power BI - for data visualization.

Project Structure

  1. Introduction and Architecture
    1. Get acquainted with the overall solution architecture and key components of the pipeline.
  2. Data Creation and Sending
    1. Write a JSON file with messages, create a Python script to send JSON objects via HTTP requests to Azure API Management.
  3. Development and Deployment of Azure Functions
    1. Learn to create and deploy Azure functions in Python using Visual Studio Code, create a function project with basic logic.
  4. Service Integration
    1. Set up and integrate Event Hubs, Azure Functions, and Cosmos DB, learn to write messages from Event Hub to Cosmos DB.
  5. Data Visualization in Power BI
    1. Connect Power BI Desktop to the Cosmos DB for real-time data visualization.

Required Knowledge and Prerequisites

  • An Azure account
  • Basic programming skills (Python)
  • Basic knowledge of working with data storage
  • Basics of API (recommended course: "Designing and Developing APIs with FastAPI")
  • Basics of working with message queues

Watch Online

This is a demo lesson (10:00 remaining)

You can watch up to 10 minutes for free. Subscribe to unlock all 14 lessons in this course and access 10,000+ hours of premium content across all courses.

View Pricing

Watch Online Data Engineering on Azure

0:00
/
#1: Data Engineering in Azure - Streaming Data Pipelines

All Course Lessons (14)

#Lesson TitleDurationAccess
1
Data Engineering in Azure - Streaming Data Pipelines Demo
02:44
2
Introduction to Datasets and Local Preprocessing
07:07
3
Deploying your Code on Visual Studio to Docker containers
05:28
4
Develop Azure Functions via Python and VS Code
05:53
5
Deploy Azure Function to Azure Function App and Test it
06:27
6
Integrate Azure Function with Blob Storage via bindings
04:59
7
Expose Azure Function as a Backend, and Test it from Insomnia
07:06
8
Securely Store Secrets in Azure Key Vault and Connect APIM to Key Vault
04:42
9
Add Basic authentication in API Management using Key Vault and Named Values
04:36
10
Test APIM and Imported Azure Function App and Function via Local Python Program
02:35
11
Create Event Hubs and Test Capture Events Feature
07:00
12
Modify Existing Azure Function to Include Event Hubs Binding and Test It
06:43
13
Create a Cosmos DB (Core SQL) and Create a New Azure Function that writes Messages to Cosmos DB
09:04
14
Connect Power Bi Desktop via Connector, and create a dashboard
06:33

Unlock unlimited learning

Get instant access to all 13 lessons in this course, plus thousands of other premium courses. One subscription, unlimited knowledge.

Learn more about subscription

Comments

0 comments

Want to join the conversation?

Sign in to comment

Similar courses

dbt for Data Engineers

dbt for Data Engineers

Sources: Andreas Kretz
dbt (data build tool) is a data transformation tool with a priority on SQL. It allows for simple and transparent transformation, testing, and documentation...
1 hour 52 minutes 55 seconds
Machine Learning A-Z : Become Kaggle Master

Machine Learning A-Z : Become Kaggle Master

Sources: udemy
Want to become a good Data Scientist? Then this is a right course for you. This course has been designed by IIT professionals who have mastered in Mathematics and Data Science....
36 hours 23 minutes 54 seconds
AZ-104 Microsoft Azure Administrator

AZ-104 Microsoft Azure Administrator

Sources: Adrian Cantrill
This course is up-to-date and corresponds to the latest version of the AZ-104 exam (as of October 26, 2023). This is a GUEST course created by James Lee.
27 hours 1 minute 23 seconds
Getting Started with Embedded AI | Edge AI

Getting Started with Embedded AI | Edge AI

Sources: udemy
Nowadays, you may have heard of many keywords like Embedded AI /Embedded ML /Edge AI, the meaning behind them is the same, I.e. To make an AI algorithm or model
3 hours 33 minutes 42 seconds
Machine Learning Design Questions

Machine Learning Design Questions

Sources: algoexpert
Several videos presenting the practice required to pass machine learning design interviews. It will not be boring.
3 hours 3 minutes 57 seconds