Data Engineering on GCP
1h 17m 33s
English
Paid
Course description
Google Cloud Platform (GCP) is one of the most popular cloud platforms in the world, providing an extensive set of tools and services for building, managing, and optimizing data pipelines. GCP enables efficient storage, processing, analysis, and visualization of data, helping data engineers create scalable and high-performance solutions.
Read more about the course
What You Will Learn in the Course
In this practical course, you will step by step create your own project on GCP:
- Extract data from an external weather API
- Process it through a pipeline using GCP cloud services
- Store the data in a server database
- Create visualizations using Looker Studio
The course will help you master GCP from scratch, and the skills you acquire will also be useful when working with other cloud platforms such as AWS, as many of the services are quite similar.
In addition to the course, you will have access to a GitHub repository with a project overview and ready-made code snippets to help you quickly replicate the training examples.
Course Structure
- Project Data and Goals
- We will analyze the pipeline architecture, define the project goals, and get acquainted with the API for obtaining weather data. You will also learn how to set up an account in Google Cloud and activate the necessary services (by the way, Google provides $300 for free platform testing!).
- Project Preparation
- Create a project in Google Cloud, activate APIs, and set up schedules for task automation.
- Pipeline Creation: Extracting Data from API
- Set up the necessary resources for pipeline operation:
- A server database MySQL via Cloud SQL
- A virtual machine based on Linux via Compute Engine for database management
- Cloud Scheduler for scheduling API calls
- Server functions for data processing
- Pub/Sub message queue for data transfer between services
- Writing Data to the Database
- Learn to write server functions to store data in MySQL, test the writing process, and ensure that data is stored correctly.
- Data Visualization
- Set up Looker Studio to create clear visualizations: build bubble charts, time series, and organize weather data monitoring.
This course will give you practical experience with Google Cloud Platform tools and help develop key skills for working as a data engineer.
Watch Online
Watch Online Data Engineering on GCP
0:00
/ #1: Introduction
All Course Lessons (23)
| # | Lesson Title | Duration | Access |
|---|---|---|---|
| 1 | Introduction Demo | 01:14 | |
| 2 | GitHub & the team | 01:31 | |
| 3 | Architecture of this project | 03:20 | |
| 4 | Introduction Weather API | 02:19 | |
| 5 | Setup Google Cloud Account | 02:13 | |
| 6 | Creating the project | 02:36 | |
| 7 | Enabling the required APIs | 01:35 | |
| 8 | Configure scheduling | 02:21 | |
| 9 | Setup VM for database interaction | 02:54 | |
| 10 | Setup mysql database | 02:17 | |
| 11 | Setup vm client and create database | 02:47 | |
| 12 | Creating pub/sub message queue | 01:42 | |
| 13 | Create cloud function to pull data form API | 04:18 | |
| 14 | Explanation code pull from API | 04:21 | |
| 15 | Create function to write to db | 07:48 | |
| 16 | Explanation code write data to db | 05:57 | |
| 17 | Testing the function | 05:52 | |
| 18 | Create function write data to db - pull | 03:54 | |
| 19 | Explanation code write data to db - pull | 04:34 | |
| 20 | Setup Looker Studio and create bubble chart | 02:21 | |
| 21 | Setup Looker Studio and create time series chart | 01:58 | |
| 22 | Pipeline Monitoring | 06:21 | |
| 23 | Conclusion & Challenges | 03:20 |
Unlock unlimited learning
Get instant access to all 22 lessons in this course, plus thousands of other premium courses. One subscription, unlimited knowledge.
Learn more about subscriptionComments
0 commentsSimilar courses
Machine Learning in JavaScript with TensorFlow.js
Sources: udemy
Interested in using Machine Learning in JavaScript applications and websites? Then this course is for you! This is the tutorial you've been looking for to becom
6 hours 42 minutes 20 seconds
Machine Learning A-Z : Become Kaggle Master
Sources: udemy
Want to become a good Data Scientist? Then this is a right course for you. This course has been designed by IIT professionals who have mastered in Mathematics and Data Science....
36 hours 23 minutes 54 seconds
Introduction to Data Engineering 2025
Sources: Andreas Kretz
This introductory course will help you better understand what data engineering is and what role a data engineer plays in the field of Data Science. To begin...
44 minutes 26 seconds
Apache Kafka Fundamentals
Sources: Andreas Kretz
In this course, you will acquire the basic knowledge necessary for confidently starting to work with Apache Kafka. You will learn how to set up a message...
1 hour 4 minutes 52 seconds
Schema Design Data Stores
Sources: Andreas Kretz
During my coaching sessions, one important topic repeatedly comes up - designing diagrams. Therefore, I decided to create a separate course in the academy to...
2 hours 30 minutes 25 seconds
Want to join the conversation?
Sign in to comment