Data Engineering on AWS
4h 46m 38s
English
Paid
Course description
This course is the perfect start for those who want to master cloud technologies and begin working with Amazon Web Services (AWS), one of the most popular platforms for data processing. The course is especially useful for beginner data engineers and those seeking their first job in this field.
Throughout the course, you will create a fully-fledged end-to-end project based on data from an online store. Step by step, you will learn to model data, build pipelines, and work with key AWS tools: Lambda, API Gateway, Kinesis, DynamoDB, Redshift, Glue, and S3.
Read more about the course
What to expect in the course:
- Data Work
- Learn the structure and types of data you will be working with. Define the project goals - an important step for successful implementation.
- Platform and Pipeline Design
- Get acquainted with the platform architecture and design pipelines: for data loading, storage in S3 (Data Lake), processing in DynamoDB (NoSQL), and Redshift (Data Warehouse). Learn to build pipelines for interfaces and data streaming.
- Basics of AWS
- Create an account in AWS, understand access and security management (IAM), get introduced to CloudWatch and the Boto3 library for working with AWS through Python.
- Data Ingestion Pipeline
- Create an API via API Gateway, send data to Kinesis, configure IAM, and develop an ingestion pipeline in Python.
- Data Transfer to S3 (Data Lake)
- Set up a Lambda function to receive data from Kinesis and save it to S3.
- Data Transfer to DynamoDB
- Implement a pipeline for transferring data from Kinesis to DynamoDB - a fast NoSQL database.
- API for Data Access
- Create an API for working with data in the database. Learn why direct access from visualization to the database is a bad practice.
- Data Visualization in Redshift
- Send streaming data to Redshift via Kinesis Firehose, create a Redshift cluster, configure security, create tables, and set up Firehose. Connect Power BI to Redshift for data analysis.
- Batch Processing: AWS Glue, S3, and Redshift
- Master batch data processing: set up and run Glue to write data from S3 to Redshift, understand Crawler and data catalog, and learn to debug processes.
This course will help you gain practical experience in creating streaming and batch pipelines in AWS, as well as mastering key tools for working with cloud data.
Watch Online
0:00
/ #1: Important: Before you start!
All Course Lessons (58)
| # | Lesson Title | Duration | Access |
|---|---|---|---|
| 1 | Important: Before you start! Demo | 00:31 | |
| 2 | Introduction | 02:22 | |
| 3 | Data Engineering | 04:16 | |
| 4 | Data Science Platform | 05:21 | |
| 5 | Data Types You Encounter | 03:04 | |
| 6 | What Is A Good Dataset | 02:55 | |
| 7 | The Dataset We Use | 03:17 | |
| 8 | Defining The Purpose | 06:28 | |
| 9 | Relational Storage Possibilities | 03:47 | |
| 10 | NoSQL Storage Possibilities | 06:29 | |
| 11 | Selecting The Tools | 03:50 | |
| 12 | Client | 03:06 | |
| 13 | Connect | 01:19 | |
| 14 | Buffer | 01:30 | |
| 15 | Process | 02:43 | |
| 16 | Store | 03:42 | |
| 17 | Visualize | 03:02 | |
| 18 | Data Ingestion Pipeline | 03:01 | |
| 19 | Stream To Raw Storage Pipeline | 02:20 | |
| 20 | Stream To DynamoDB Pipeline | 03:10 | |
| 21 | Visualization API Pipeline | 02:57 | |
| 22 | Visualization Redshift Data Warehouse Pipeline | 05:30 | |
| 23 | Batch Processing Pipeline | 03:20 | |
| 24 | Create An AWS Account | 01:59 | |
| 25 | Things To Keep In Mind | 02:46 | |
| 26 | IAM Identity & Access Management | 04:08 | |
| 27 | Logging | 02:23 | |
| 28 | AWS Python API Boto3 | 02:58 | |
| 29 | Development Environment | 04:03 | |
| 30 | Create Lambda for API | 02:34 | |
| 31 | Create API Gateway | 08:31 | |
| 32 | Setup Kinesis | 01:39 | |
| 33 | Setup IAM for API | 05:01 | |
| 34 | Create Ingestion Pipeline (Code) | 06:10 | |
| 35 | Create Script to Send Data | 05:47 | |
| 36 | Test The Pipeline | 04:54 | |
| 37 | Setup S3 Bucket | 03:43 | |
| 38 | Configure IAM For S3 | 03:22 | |
| 39 | Create Lambda For S3 Insert | 07:17 | |
| 40 | Test The Pipeline | 04:02 | |
| 41 | Setup DynamoDB | 09:01 | |
| 42 | Setup IAM For DynamoDB Stream | 03:37 | |
| 43 | Create DynamoDB Lambda | 09:21 | |
| 44 | Create API & Lambda For Access | 06:11 | |
| 45 | Test The API | 04:48 | |
| 46 | Setup Redshift Data Warehouse | 08:09 | |
| 47 | Security Group For Firehose | 03:13 | |
| 48 | Create Redshift Tables | 05:52 | |
| 49 | S3 Bucket & jsonpaths.json | 03:03 | |
| 50 | Configure Firehose | 07:59 | |
| 51 | Debug Redshift Streaming | 07:44 | |
| 52 | Bug-fixing | 05:59 | |
| 53 | Power Bi | 12:17 | |
| 54 | AWS Glue Basics | 05:15 | |
| 55 | Glue Crawlers | 13:10 | |
| 56 | Glue Jobs | 13:44 | |
| 57 | Redshift Insert & Debugging | 07:17 | |
| 58 | What We Achieved & Improvements | 10:41 |
Unlock unlimited learning
Get instant access to all 57 lessons in this course, plus thousands of other premium courses. One subscription, unlimited knowledge.
Learn more about subscriptionComments
0 commentsWant to join the conversation?
Sign in to commentSimilar courses
Becoming a Better Data Engineer
Sources: Andreas Kretz
Data engineering is not just about moving information from one place to another. It is about creating reliable, scalable, and efficient systems that...
1 hour 46 minutes 10 seconds
Getting Started with Embedded AI | Edge AI
Sources: udemy
Nowadays, you may have heard of many keywords like Embedded AI /Embedded ML /Edge AI, the meaning behind them is the same, I.e. To make an AI algorithm or model
3 hours 33 minutes 42 seconds
Complete Machine Learning and Data Science: Zero to Mastery
Sources: udemy, zerotomastery.io
This is a brand new Machine Learning and Data Science course just launched January 2020 and updated this month with the latest trends and skills! Become a complete Data Scientis...
43 hours 22 minutes 23 seconds
AWS Certified SysOps Administrator - Associate
Sources: Adrian Cantrill
The AWS Certified Sysops Administrator Associate is one of the most valuable and in-demand cloud certifications, part of the AWS Associate (All-3) series. It covers all of the c...
Build Fast Masterclass
Sources: BuildFast Academy
How to finally Launch your AI Product (without ripping your hair out).. that makes you money in 30 days (or less). But unlike other AI courses, you won't learn
7 hours 22 minutes 11 seconds