Data Engineering on GCP

1h 17m 33s
English
Paid
Google Cloud Platform (GCP) is one of the most popular cloud platforms in the world, providing an extensive set of tools and services for building, managing, and optimizing data pipelines. GCP enables efficient storage, processing, analysis, and visualization of data, helping data engineers create scalable and high-performance solutions.
Read more about the course

What You Will Learn in the Course

In this practical course, you will step by step create your own project on GCP:

  • Extract data from an external weather API
  • Process it through a pipeline using GCP cloud services
  • Store the data in a server database
  • Create visualizations using Looker Studio

The course will help you master GCP from scratch, and the skills you acquire will also be useful when working with other cloud platforms such as AWS, as many of the services are quite similar.

In addition to the course, you will have access to a GitHub repository with a project overview and ready-made code snippets to help you quickly replicate the training examples.

Course Structure

  • Project Data and Goals
    • We will analyze the pipeline architecture, define the project goals, and get acquainted with the API for obtaining weather data. You will also learn how to set up an account in Google Cloud and activate the necessary services (by the way, Google provides $300 for free platform testing!).
  • Project Preparation
    • Create a project in Google Cloud, activate APIs, and set up schedules for task automation.
  • Pipeline Creation: Extracting Data from API
  • Set up the necessary resources for pipeline operation:
    • A server database MySQL via Cloud SQL
    • A virtual machine based on Linux via Compute Engine for database management
    • Cloud Scheduler for scheduling API calls
    • Server functions for data processing
    • Pub/Sub message queue for data transfer between services
  • Writing Data to the Database
    • Learn to write server functions to store data in MySQL, test the writing process, and ensure that data is stored correctly.
  • Data Visualization
    • Set up Looker Studio to create clear visualizations: build bubble charts, time series, and organize weather data monitoring.

This course will give you practical experience with Google Cloud Platform tools and help develop key skills for working as a data engineer.

Watch Online Data Engineering on GCP

Join premium to watch
Go to premium
# Title Duration
1 Introduction 01:14
2 GitHub & the team 01:31
3 Architecture of this project 03:20
4 Introduction Weather API 02:19
5 Setup Google Cloud Account 02:13
6 Creating the project 02:36
7 Enabling the required APIs 01:35
8 Configure scheduling 02:21
9 Setup VM for database interaction 02:54
10 Setup mysql database 02:17
11 Setup vm client and create database 02:47
12 Creating pub/sub message queue 01:42
13 Create cloud function to pull data form API 04:18
14 Explanation code pull from API 04:21
15 Create function to write to db 07:48
16 Explanation code write data to db 05:57
17 Testing the function 05:52
18 Create function write data to db - pull 03:54
19 Explanation code write data to db - pull 04:34
20 Setup Looker Studio and create bubble chart 02:21
21 Setup Looker Studio and create time series chart 01:58
22 Pipeline Monitoring 06:21
23 Conclusion & Challenges 03:20

Similar courses to Data Engineering on GCP

Data Analysis for Beginners: Excel & Pivot Tables

Data Analysis for Beginners: Excel & Pivot Tableszerotomastery.io

Category: Data processing and analysis
Duration 2 hours 10 minutes 21 seconds
Statistics Bootcamp (with Python): Zero to Mastery

Statistics Bootcamp (with Python): Zero to Masteryzerotomastery.io

Category: Python, ChatGPT, Data processing and analysis
Duration 20 hours 50 minutes 51 seconds
PyTorch for Deep Learning

PyTorch for Deep Learningzerotomastery.io

Category: Data processing and analysis
Duration 52 hours 27 seconds
Deep Learning A-Z™: Hands-On Artificial Neural Networks

Deep Learning A-Z™: Hands-On Artificial Neural Networksudemy

Category: Python, Data processing and analysis
Duration 22 hours 36 minutes 30 seconds
Platform & Pipeline Security

Platform & Pipeline SecurityAndreas Kretz

Category: Data processing and analysis
Duration 34 minutes 46 seconds
Build Fast Masterclass

Build Fast MasterclassBuildFast Academy

Category: Python, Data processing and analysis
Duration 7 hours 22 minutes 11 seconds
Data Platform & Pipeline Design

Data Platform & Pipeline DesignAndreas Kretz

Category: Data processing and analysis
Duration 1 hour 59 minutes 5 seconds
Data Analysis for Beginners: Python & Statistics

Data Analysis for Beginners: Python & Statisticszerotomastery.io

Category: Python, Data processing and analysis
Duration 6 hours 34 minutes 20 seconds
Time Series Analysis, Forecasting, and Machine Learning

Time Series Analysis, Forecasting, and Machine Learningudemy

Category: Python, Data processing and analysis
Duration 22 hours 47 minutes 45 seconds
Learning Apache Spark

Learning Apache SparkAndreas Kretz

Category: Data processing and analysis
Duration 1 hour 44 minutes 4 seconds