Data Engineering on Azure

What you will learn in the course

In this course guided by Kristian Bakarich, you will practically create a streaming data processing pipeline in Azure. As part of the project, you will learn to use key Azure services for processing Twitter data streams in JSON format, including:

• APIM (API Management) - for data intake,
• Blob Storage - for storage,
• Azure Functions - for processing,
• Cosmos DB - for storing processed data,
• Power BI - for data visualization.

Project Structure

1. Introduction and Architecture
   1. Get acquainted with the overall solution architecture and key components of the pipeline.
2. Data Creation and Sending
   1. Write a JSON file with messages, create a Python script to send JSON objects via HTTP requests to Azure API Management.
3. Development and Deployment of Azure Functions
   1. Learn to create and deploy Azure functions in Python using Visual Studio Code, create a function project with basic logic.
4. Service Integration
   1. Set up and integrate Event Hubs, Azure Functions, and Cosmos DB, learn to write messages from Event Hub to Cosmos DB.
5. Data Visualization in Power BI
   1. Connect Power BI Desktop to the Cosmos DB for real-time data visualization.

Required Knowledge and Prerequisites

• An Azure account
• Basic programming skills (Python)
• Basic knowledge of working with data storage
• Basics of API (recommended course: "Designing and Developing APIs with FastAPI")
• Basics of working with message queues