CF

Storing & Visualizing Time Series Data

2h 11m 34s
English
Paid

Enhance your skills in managing time series data with this comprehensive course. The processing, storage, and visualization of time series data are becoming increasingly important. With IoT data, system logs, and production process statistics, the volume of information needing management is constantly expanding. This course will guide you in building a complete pipeline for handling time series data effectively.

Course Overview

Time series storage systems like InfluxDB and visualization tools such as Grafana are essential for data management and analysis. This course aims to equip you with the skills to implement these technologies in practice.

Learning Outcomes

Throughout this course, you will gain practical skills such as:

  • Processing time series data from CSV files, including air quality datasets
  • Connecting to external APIs to retrieve real-time weather data
  • Writing data to InfluxDB 2.0 and querying it using Python and the Flux language
  • Creating and configuring dashboards in Grafana: including server installation, panel configuration, and access management

Course Structure

Introduction

Begin with an overview of the project, and get familiar with the dataset and the InfluxDB interface, including graphs, queries, and data structure. You'll also explore the platform architecture to understand the interaction between components.

Data Schema Design

Learn about the features of relational and time series databases and determine the appropriate scenarios for each. You'll develop skills in designing a storage schema tailored to data characteristics and usage methods.

Environment Setup

Install and launch InfluxDB and Grafana using Docker. Configure the Python library for InfluxDB, create an access token, and set up your development environment in Visual Studio Code.

Working with Test Data

Gain experience in loading test CSV files and weather data into InfluxDB using Python. Discover potential issues during data loading and learn how to resolve them efficiently.

Working with Air Quality Data

Load air quality data into InfluxDB, write queries using Python, and connect Grafana to InfluxDB for visualization. Set up a data source and create a dashboard dedicated to air quality analysis.

Working with External Weather API

Get acquainted with a weather API, obtain an access key, and manage time zones effectively. Connect to an external API, load data into InfluxDB, and visualize it on a Grafana dashboard.

Working with Grafana Dashboards

Delve into the capabilities of Grafana, including multi-user mode, access rights management, and settings for users and organizations. Conclude the project by creating two organizations in Grafana and connecting weather and air quality data, applying your knowledge in practical scenarios.

This course provides essential practical skills for modern analytics and monitoring projects, focusing on time series data storage and visualization tools.

Additional

Link to the GitHub: https://github.com/team-data-science/timeseries-data

About the Author: Andreas Kretz

Andreas Kretz thumbnail

Andreas Kretz is a German data engineer and one of the most widely followed independent voices on data engineering as a career discipline. He runs the Plumbers of Data Science brand and has been publishing tutorial material continuously since the field consolidated around the modern lake-house stack (Spark, Kafka, Snowflake, Databricks, Airflow).

His CourseFlix listing is the largest single-author catalog under this source — over thirty courses spanning data-pipeline construction, streaming architectures, the cloud-native data stack on AWS / Azure / GCP, the Python and Scala tooling that dominates the field, and the soft-skills / career side of breaking into data engineering. Material is paid and aimed at engineers transitioning into data work or already-working data engineers picking up specific tools.

Watch Online 25 lessons

This is a demo lesson (10:00 remaining)

You can watch up to 10 minutes for free. Subscribe to unlock all 25 lessons in this course and access 10,000+ hours of premium content across all courses.

View Pricing
0:00
/
#1: Introduction
All Course Lessons (25)
#Lesson TitleDurationAccess
1
Introduction Demo
03:18
2
What we are going to build
02:24
3
The data set we use
02:28
4
Relational DBs vs Time Series DBs features
06:13
5
Access patterns
02:59
6
InfluxDB key concepts
05:20
7
Schema design relational database
05:16
8
Schema design InfluxDB
03:39
9
InfluxDB & Grafana Docker setup
05:54
10
Container startup & Python lib installation
03:15
11
InfluxDB Python token & VS Code setup
01:49
12
Writing test data to influxDB with Python
08:42
13
Exploring & soliving the data type problem
05:56
14
Writing in air quality data to InfluxDB
07:33
15
Query data with Python from InfluxDB
05:38
16
Grafana data source setup
05:07
17
Create Grafana dashboard for InfluxDB
05:23
18
Weather API introduction
03:30
19
Managing time zones
06:15
20
API ingestion script
12:25
21
Grafana user & rights management
04:51
22
Beijing organization setup with direct user permissions
10:01
23
Beijing organization with team permissions
03:35
24
Weather API organization setup
07:43
25
Summary
02:20
Unlock unlimited learning

Get instant access to all 24 lessons in this course, plus thousands of other premium courses. One subscription, unlimited knowledge.

Learn more about subscription

Related courses

  • Apache Spark Certification Training thumbnailUpdated 2y ago

    Apache Spark Certification Training

    By: Florian Roscheck
    Apache Spark is a core data skill - here is how to show you got it! Learn Apache Spark from the ground up and show off your knowledge with the Databricks Associ
    15h 13m
  • Choosing Data Stores thumbnailUpdated 11mo ago

    Choosing Data Stores

    By: Andreas Kretz
    One of the key tasks when creating a data platform and pipelines is the selection of appropriate data storage systems. This course is dedicated to that topic.
    1h 25m5/5
  • Snowflake for Data Engineers thumbnailUpdated 11mo ago

    Snowflake for Data Engineers

    By: Andreas Kretz
    Embrace the future of data management and analysis with Snowflake, a revolutionary cloud-based data storage solution.
    2h 4m5/5

Frequently asked questions

What are the prerequisites for enrolling in this course?
The course does not specify any prerequisites explicitly, but a basic understanding of databases and familiarity with Python programming would be beneficial. The course involves working with InfluxDB, Grafana, and APIs, so prior exposure to these or similar tools may enhance your learning experience.
What will I be able to build after completing the course?
Upon completing the course, you will be able to build a complete pipeline for managing time series data. This includes processing data from CSV files, connecting to APIs for real-time data retrieval, writing and querying data in InfluxDB 2.0, and creating dashboards in Grafana for data visualization.
Who is the target audience for this course?
This course is designed for data analysts, engineers, and developers who are interested in enhancing their skills in managing and visualizing time series data. It is particularly beneficial for those working with IoT data, system logs, or production process statistics.
How does this course compare in scope to other data management courses?
This course focuses specifically on time series data management and visualization, using tools like InfluxDB and Grafana. Unlike general data management courses, it offers a specialized look at handling time series data, covering topics such as schema design for time series databases and API integration for real-time data.
Which platforms and tools are specifically covered in this course?
The course covers InfluxDB 2.0 and Grafana as the primary tools for time series data management and visualization. It also involves using Python for data processing and querying, and Docker for setting up the InfluxDB and Grafana environments.
What topics are not covered in this course?
The course does not cover general database management or visualization techniques outside of the context of time series data. It also does not delve into machine learning or predictive analytics, focusing instead on data handling and real-time visualization.
What is the estimated time commitment for this course?
The course consists of 25 lessons, each focusing on different aspects of time series data management and visualization. While the total runtime is not specified, students should plan to spend additional time on hands-on exercises and reviewing course material to fully grasp the concepts.