Name: Data Engineering on GCP
Price: 9 USD
Availability: InStock

Data Engineering on GCP

Google Cloud Platform (GCP) is one of the most popular cloud platforms in the world, providing an extensive set of tools and services for building, managing, and optimizing data pipelines. GCP enables efficient storage, processing, analysis, and visualization of data, helping data engineers create scalable and high-performance solutions.

What You Will Learn in the Course

In this practical course, you will step by step create your own project on GCP:

Extract data from an external weather API
Process it through a pipeline using GCP cloud services
Store the data in a server database
Create visualizations using Looker Studio

The course will help you master GCP from scratch, and the skills you acquire will also be useful when working with other cloud platforms such as AWS, as many of the services are quite similar.

In addition to the course, you will have access to a GitHub repository with a project overview and ready-made code snippets to help you quickly replicate the training examples.

Course Structure

Project Data and Goals

We will analyze the pipeline architecture, define the project goals, and get acquainted with the API for obtaining weather data. You will also learn how to set up an account in Google Cloud and activate the necessary services. (By the way, Google provides $300 for free platform testing!)

Project Preparation

Create a project in Google Cloud, activate APIs, and set up schedules for task automation.

Pipeline Creation: Extracting Data from API

Set up the necessary resources for pipeline operation:

A server database MySQL via Cloud SQL
A virtual machine based on Linux via Compute Engine for database management
Cloud Scheduler for scheduling API calls
Server functions for data processing
Pub/Sub message queue for data transfer between services

Writing Data to the Database

Learn to write server functions to store data in MySQL, test the writing process, and ensure that data is stored correctly.

Data Visualization

Set up Looker Studio to create clear visualizations: build bubble charts, time series, and organize weather data monitoring.

This course will give you practical experience with Google Cloud Platform tools and help develop key skills for working as a data engineer.

About the Author: Andreas Kretz

I am a senior data engineer and trainer, a tech enthusiast, and a father. For more than ten years, I have been passionate about Data Engineering. Initially, I became a self-taught data engineer and then led a team of data engineers at a large company. When I realized the great demand for education in this field, I followed my passion and founded my own Data Engineering Academy. Since then, I have helped over 2,000 students achieve their goals.

Watch Online 23 lessons

0:00

#1: Introduction

All Course Lessons (23)

#	Lesson Title	Duration
1	Introduction Demo	01:14
2	GitHub & the team	01:31
3	Architecture of this project	03:20
4	Introduction Weather API	02:19
5	Setup Google Cloud Account	02:13
6	Creating the project	02:36
7	Enabling the required APIs	01:35
8	Configure scheduling	02:21
9	Setup VM for database interaction	02:54
10	Setup mysql database	02:17
11	Setup vm client and create database	02:47
12	Creating pub/sub message queue	01:42
13	Create cloud function to pull data form API	04:18
14	Explanation code pull from API	04:21
15	Create function to write to db	07:48
16	Explanation code write data to db	05:57
17	Testing the function	05:52
18	Create function write data to db - pull	03:54
19	Explanation code write data to db - pull	04:34
20	Setup Looker Studio and create bubble chart	02:21
21	Setup Looker Studio and create time series chart	01:58
22	Pipeline Monitoring	06:21
23	Conclusion & Challenges	03:20

Unlock unlimited learning

Get instant access to all 22 lessons in this course, plus thousands of other premium courses. One subscription, unlimited knowledge.

Learn more about subscription