Data engineers often need to quickly set up a simple ETL script that just gets the job done. In this project, you will learn how to easily implement such an ETL on AWS: connect live data from a weather API and write it to a TDengine time-series database.
Course Overview
Embark on a journey to master the integration of Docker, AWS, TDengine, and Grafana for efficient ETL processes. This course provides hands-on experience with cutting-edge technologies to streamline data engineering tasks.
Learning Objectives
The Basics of Temporal Databases
You will get acquainted with the basics of working with temporal databases, their architecture, and use cases.
Working with a Public Weather API
Learn to set up and explore an external weather API, and write a Python script to read real-time data from the API.
Docker ETL on AWS
Discover how to package the script into a Docker container and deploy it as a serverless ETL using Amazon Elastic Container Registry (ECR), Lambda, and EventBridge.
TDengine Setup
Get familiar with TDengine, set up an instance via the TDengine Cloud, and configure the database for optimum performance.
Data Visualization in Grafana
Learn how to visualize data from the API stored in TDengine using Grafana. Connect TDengine to Grafana and create a comprehensive dashboard for data analysis.
Course Benefits
- Hands-on experience with real-world data integration and visualization.
- Skills to implement Dockerized ETL projects on the AWS cloud.
- Profound understanding of temporal databases and their applications.
- Ability to leverage Grafana for impactful data visualization.
