Data Platform & Pipeline Design

Basic Module

Fundamentals of Platforms and Pipelines

You will get acquainted with platform architectures and different types of pipelines. You will learn how they differ, how they work, what a machine learning pipeline looks like, and how to integrate them within a single system.

Platform Architecture and End-to-End Pipeline

You will understand the structure of a typical platform architecture: connection, buffering, processing, storage, and data visualization. By examining an end-to-end pipeline, you will learn how to apply this structure in your work.

Push and Pull Pipelines

You will understand the difference between the push and pull model of data transmission—sending versus fetching. Includes illustrative examples and diagrams.

Batch and Streaming Pipelines

This is one of the most important blocks for a data engineer. You will learn to distinguish and apply batch and streaming processing depending on the scenario.

Data Streams Visualization

You will understand how to visualize data processing and storage—even if you don't have direct access to them. An example with Apache Spark will help reinforce the material.

Lambda Architecture

You will learn how batch and stream pipelines are integrated within a single platform—especially important for ML, where training is done on batch data and application through streaming.

Platform Examples

You will study architecture templates on AWS, GCP, Azure, and Hadoop, where you will see how tools like Lambda, API Gateway, and DynamoDB fit into the real infrastructure.

Advanced Module

Processing Models: Event-Driven, Batch, and Stream

You will understand the differences between event-driven, batch, micro-batching, and streaming. Learn how to choose the appropriate processing type for tasks: analytics, transactions, reverse ETL, and more.

Targeted Design and Platform Schema Replication

You will revisit the platform schema and learn to align business goals and data types with architectural solutions. Instead of choosing tools "by feel," you will learn to design the system from the task.

Modern Architectures: Lakehouse and Medallion

You will learn how Lakehouse combines file storage and transactional tables, and how bronze-silver-gold layers in the Medallion architecture help maintain order and scalability.

Machine Learning and Generative AI (GenAI)

You will learn how machine learning pipelines integrate into the platform: where training, inference, and deployment occur. Get acquainted with the concepts of semantic search and Retrieval-Augmented Generation (RAG)—the foundation of modern AI applications.

Platform Testing

A brief but important module: testing strategies for pipelines at all stages—from loading and processing to data transformation.

This course will give you a comprehensive understanding of platforms and pipelines and will teach you how to build efficient architecture applicable in real cloud solutions. It is ideal for both beginner engineers and those who want to advance to the next level.