Choosing the right data storage is a fundamental task in creating a data platform and building pipelines. This course is dedicated to guiding you through this important topic.
Overview of Data Storage Types
Throughout this course, we will explore different types of data storage solutions including relational and NoSQL databases, as well as data warehouses and data lakes. You will learn the appropriate scenarios for using each type of storage and how to effectively integrate them into your architecture.
Completing this course will equip you with the necessary knowledge to understand various data storage solutions and help you make informed decisions, enhancing your capabilities as a data engineer. Future courses will dive deeper into specific technologies within each category.
Basics of Data Warehousing
We begin by discussing foundational principles: the key differences between OLTP (Operational Transactional Systems) and OLAP (Analytical Systems), and their respective use cases. You will also learn about ETL (Extract, Transform, Load) and ELT (Extract, Load, Transform), and how these processes influence the selection of data warehouses. At the end of this section, additional resources will be provided for further exploration and comparison of different data warehouse types.
Understanding Relational Databases
This section offers a step-by-step guide to selecting a suitable data storage solution for your projects. You will gain an in-depth understanding of relational databases, including CRUD operations and the ACID principles, along with examples from specific Database Management Systems (DBMS).
Exploring NoSQL Databases
In this part of the course, you will discover what NoSQL databases are, including various types such as document-based, columnar, temporal, and search databases. We will examine their unique characteristics and suitable applications. Additionally, we will discuss the trade-offs between read and write speeds and emphasize the importance of setting goals when selecting storage solutions.
Data Warehouses vs. Data Lakes
The course concludes with a detailed comparison of data warehouses and data lakes. You will learn about the differences between these storage solutions and the specific use cases where each excels. This knowledge will further aid in your decision-making process when architecting data platforms.
