Data Engineering on GCP

What You Will Learn in the Course

In this practical course, you will step by step create your own project on GCP:

• Extract data from an external weather API
• Process it through a pipeline using GCP cloud services
• Store the data in a server database
• Create visualizations using Looker Studio

The course will help you master GCP from scratch, and the skills you acquire will also be useful when working with other cloud platforms such as AWS, as many of the services are quite similar.

In addition to the course, you will have access to a GitHub repository with a project overview and ready-made code snippets to help you quickly replicate the training examples.

Course Structure

• Project Data and Goals
  • We will analyze the pipeline architecture, define the project goals, and get acquainted with the API for obtaining weather data. You will also learn how to set up an account in Google Cloud and activate the necessary services (by the way, Google provides $300 for free platform testing!).
• Project Preparation
  • Create a project in Google Cloud, activate APIs, and set up schedules for task automation.
• Pipeline Creation: Extracting Data from API
• Set up the necessary resources for pipeline operation:
  • A server database MySQL via Cloud SQL
  • A virtual machine based on Linux via Compute Engine for database management
  • Cloud Scheduler for scheduling API calls
  • Server functions for data processing
  • Pub/Sub message queue for data transfer between services
• Writing Data to the Database
  • Learn to write server functions to store data in MySQL, test the writing process, and ensure that data is stored correctly.
• Data Visualization
  • Set up Looker Studio to create clear visualizations: build bubble charts, time series, and organize weather data monitoring.

This course will give you practical experience with Google Cloud Platform tools and help develop key skills for working as a data engineer.