CF

Data Engineering Design Patterns

13h 15m 48s
English
Paid

Data Engineering Design Patterns is a 60-lesson 13 hours 15 minutes self-paced course by Joseph Machado. Enhance your data engineering skills by mastering key design patterns that underpin reliable, scalable, and professionally designed data systems.

Course facts

Lessons
60
Duration
13 hours 15 minutes
Level
All levels
Language
English
Updated
Instructor
Joseph Machado
Price
Premium

Enhance your data engineering skills by mastering key design patterns that underpin reliable, scalable, and professionally designed data systems.The course combines theory, practice, live sessions, and a large final project, allowing you to transition from a chaotic to a systematic approach in data work.

Course Overview

The program consists of 61 lessons and covers a complete set of architectural patterns used in modern data engineering. You will learn approaches that help build predictable, idempotent, scalable, and easily maintainable pipelines and prepare for architectural interviews at leading companies.

Who This Course Is For

Target Audience

The course will be particularly beneficial for Data Engineers, Analytics Engineers, data developers, and architects who want to:

  • Move from a set of scripts to systematic design of architecture.
  • Build fault-tolerant and scalable pipelines.
  • Confidently pass System Design interviews and architectural sections.

Course Program

Data Warehousing Design

Key Topics

Fundamental approaches to analytical modeling and warehouse design.

  • Kimball modeling: dimension and fact types, best practices.
  • Using Python as an ETL management layer.
  • Partitioning and bucketing for storage optimization.
  • Bus Matrix for requirement alignment.
  • Incremental and full-refresh pipelines.

Data Pipeline Design

Patterns for increasing stability, predictability, and fault tolerance.

  • Separation of responsibility between management and execution layers.
  • Implementation of SCD2 using MERGE INTO.
  • Pipeline design considering Backfill.
  • Handling Late-arriving events.
  • Idempotency and self-recovery of pipelines.

Medallion Architecture Streaming

Standardization and formalization of data flows within a company.

  • 3-hop architecture: Bronze, Silver, Gold.
  • Designing wide OBT tables with nested structures.
  • Lambda approach: frequent and corrective starts.

Data Quality

Practices for increasing data trust and minimizing noisy alerts.

  • WAP (Write-Audit-Publish) pattern.
  • Choosing effective types of checks.
  • Integrating Data Quality into production.

Scheduling and Orchestration (Airflow)

Building managed pipelines with predictable dependencies.

  • Pattern "1 DAG = 1 Output".
  • Passing time ranges for incremental loading.
  • Data-driven scheduling.
  • Airflow architecture under the hood.

Code Testing

Approaches for increasing system reliability and predictability.

  • Unit tests with Pytest.
  • Integration testing of adjacent systems.

Data Contracts

A modern mechanism for formalizing interaction between data suppliers and consumers.

  • Defining data structure and quality requirements at the system junction.

Capstone Project

The final module where you will create a complete architecture and production pipeline according to industry standards.

  • Designing architecture and data flows.
  • Building pipelines for facts and dimensions.
  • Implementing Data Quality checks.
  • Orchestrating processes and visualizing results.
  • Project presentation focusing on engineering expertise.

Interview Preparation

Step-by-step methodology for preparing for a Data Engineer position interview.

  • Algorithms and data structures.
  • Advanced SQL.
  • System Design, data modeling, metrics.
  • Behavioral interview using STAR.
  • Preparation strategy for specific companies.

Who teaches Data Engineering Design Patterns? Joseph Machado

Joseph Machado thumbnail

After more than 15 years of developing distributed data platforms processing exabytes of information, I've spent countless hours sifting through official documentation, little-known blogs, and trial and error—all for the sake of simply grasping best practices and real technological trade-offs.

Most educational resources lacked depth. They were either optimized for SEO promotion or sponsored by specific vendors who only spoke about the advantages of their products.

I created the project Start Data Engineering to address this. My goal is to provide engineers with quality, project-based, and accessible content that truly prepares them for the tough realities of the working world. Over 20,000 professionals have already used these guides to enhance their skills.

Three main principles of my courses:

  • Practical applicability: You gain knowledge that can be applied in your work right away, rather than just reading and forgetting.
  • Code-first approach: All lessons are built on practice, forcing you to learn through real action.
  • Honest assessment of trade-offs: No blind faith in tools—I explain all the pros and cons so you clearly understand when and which technology to apply.

What lessons are included in Data Engineering Design Patterns?

This is a demo lesson (10:00 remaining)

You can watch up to 10 minutes for free. Subscribe to unlock all 60 lessons in this course and access 10,000+ hours of premium content across all courses.

View Pricing
0:00
/
#1: Welcome
All Course Lessons (60)
#Lesson TitleDurationAccess
1
Welcome Demo
02:17
2
Course Objective & Code
04:53
3
Data Overview
02:33
4
Office Hours 2026-05-05
01:48:48
5
Office Hours 2026-05-07
01:35:25
6
Office Hours 2026-05-12
01:26:33
7
Office Hours 2026-05-14
01:30:30
8
Office Hours 2026-05-19
01:09:18
9
What is a Data Warehouse
09:23
10
Kimball Data Model
06:27
11
Analytical queries involve joining fact and dimension tables and grouping by dimension attribute(s)
03:49
12
Facts are generated by your system, the user's browser, or purchased from a third-party
05:16
13
Dimensions represent a business concept
12:29
14
Python for extracting, transforming, & loading data into a modelled destination
10:36
15
Data Storage Patterns Partitioning for efficient reads & Bucketing for efficient joinsgroup by
09:08
16
Bus Matrix Get everyone on the same page
03:50
17
Pipeline Types Full refresh processes the entire source, and incremental processes a time range-specific source
06:12
18
Recap
01:10
19
Python enables you to control multiple systems
03:00
20
Create SCD2 tables with MERGE INTO
10:25
21
Backfills are inevitable, design your pipelines for them
07:06
22
Wait to process the fact data until you are certain most of it has arrived
06:48
23
Data pipeline scripts should be re-runnable without creating duplicate or partial data (aka idempotent)
05:28
24
Self-healing pipelines make maintenance easy.
04:10
25
Recap
01:39
26
3-hop architecture Bronze is source, Silver is factdims, & Gold is summary tables
06:41
27
Gold tables are for select from gold_tbl by end users
09:15
28
Use nested data structures to create wide OBTs
09:11
29
Run fact pipelines hourly for data availability and daily to catch late events (aka Lambda Architecture)
05:45
30
Recap
00:53
31
Check your data before end-users use it, with the WAP pattern
04:43
32
Choose the type of data quality check based on the data
06:17
33
Implementing DQ checks
09:13
34
Recap
01:11
35
Scheduling data pipelines with Apache Airflow
13:42
36
Time range of data to be processed is supplied by Airflow
11:20
37
Running pipeline when a dataset is updated
10:46
38
Airflow Architecture
04:43
39
Recap
02:10
40
Check that your code does what you think it does with tests
11:02
41
Use Pytest to manage tests
11:48
42
Ensure systems work together as expected with Integration tests
04:49
43
Recap
01:07
44
Data contract defines your requirements
08:06
45
Objective
03:48
46
Define Outcomes
05:53
47
Architecture & Data Flow
04:27
48
Write code - Bronze & Silver
07:58
49
Write code - Gold
07:25
50
Data quality
03:42
51
Visualizing outputs
03:45
52
Orchestrate your pipelines & Present them
05:13
53
Recap
02:11
54
Interview Prep Is a Process, Not a Checklist
00:52
55
Data Structures & Algorithms
17:42
56
SQL & Data Manipulation
09:54
57
System Design, Defining Metrics & Data Modeling
13:54
58
Behavioral Interview
04:56
59
Company Specific Preparation
02:18
60
Recap
01:55
Unlock unlimited learning

Get instant access to all 59 lessons in this course, plus thousands of other premium courses. One subscription, unlimited knowledge.

Learn more about subscription

What courses are similar to Data Engineering Design Patterns?

More courses by Joseph Machado

  • Spark for Data Engineers thumbnailNew

    Spark for Data Engineers

    Practical course on Apache Spark for confident work with distributed data processing, pipeline optimization, and creating efficient Big Data solutions.
    4h 28m

Frequently asked questions

What are the prerequisites for enrolling in the Data Engineering Design Patterns course?
The course is designed for professionals such as Data Engineers, Analytics Engineers, data developers, and architects. It is beneficial to have a background in data engineering or a related field to fully grasp the advanced design patterns discussed. Familiarity with Python, SQL, and basic data warehousing concepts is recommended to gain the most from the course content.
What projects or systems will I be able to build after completing this course?
Upon completing the course, students will be equipped to design and implement fault-tolerant and scalable data pipelines. They will learn to apply architectural patterns like the 3-hop architecture, implement SCD2 tables using MERGE INTO, and schedule data pipelines using Apache Airflow. The course emphasizes creating predictable, idempotent pipelines that ensure data integrity and reliability.
How does this course compare in depth and scope to other data engineering courses?
This course offers a comprehensive exploration of architectural design patterns in data engineering, focusing on building scalable and maintainable systems. It covers fundamental approaches like Kimball modeling and advanced topics such as partitioning, bucketing, and self-healing pipelines. Unlike some introductory courses, this program is tailored for those looking to deepen their understanding and prepare for system design interviews at leading companies.
What specific tools and platforms are taught in this course?
The course extensively uses Python as an ETL management layer and covers the use of Apache Airflow for scheduling data pipelines. Students will also learn to implement data warehousing concepts using Kimball modeling and strategies for data quality checks. The focus is on practical tools and techniques that enhance data system reliability and scalability.
What topics are not covered in this course?
The course does not cover introductory data engineering topics or general programming fundamentals. It assumes a baseline knowledge of data systems and focuses on advanced design patterns and architectural strategies. It also does not delve into specific cloud platforms or vendor-specific technologies, maintaining a focus on overarching design principles.
What is the expected time commitment for this course?
The course consists of 61 lessons, including theoretical lectures and practical exercises. In addition to the lesson runtime, students should allocate time for the final project and live sessions, which are integral to mastering the content. While the exact time commitment will vary by individual, students should expect to dedicate several hours per week to fully engage with the course material.
How can the skills learned in this course be applied to other careers or courses?
The design patterns and architectural knowledge gained in this course are applicable to various roles within data engineering and beyond. Skills such as creating idempotent pipelines, implementing data quality checks, and using architectural frameworks are valuable in roles focused on data analytics, data warehousing, and system architecture. Additionally, the preparation for architectural interviews enhances readiness for positions at leading tech companies.