Databricks is one of the most popular platforms for data processing using Apache Spark and for creating modern data warehouses (Lakehouse). In this course, you will learn everything you need for a confident start with Databricks, from the basics of the platform to creating your own pipelines and connecting BI tools.
You will learn how Databricks works, why to use it, create your notebooks, set up a computing cluster, and get acquainted with Databricks SQL Warehouse.
1. Installation and Data Preparation
Before getting started with practical work, you will set up Databricks on AWS, create an S3 bucket for data storage, and set up a workspace. You will also examine the AWS CloudFormation template that Databricks uses to understand how the infrastructure is automatically deployed.
You will review the created cluster and become familiar with the dataset on which you will build your ETL process.
2. Practice: Data Processing
You will learn two ways to load data into Databricks: directly or through S3 followed by integration. You will also learn how to create code repositories. This can be done in two ways: connect a GitHub repository or create a repository manually right in Databricks.
During the project, you will complete two key tasks:
You will also learn how data is stored within Databricks.
3. Data Warehouse and External Integrations
Finally, you will connect Power BI to Databricks and try both integration methods: through a compute cluster and through SQL Warehouse. This way, you will learn how to integrate Databricks with external analytics tools.
Recommendations Before Starting
Before starting this course, it is recommended to complete the "Apache Spark Basics" course. With these foundational skills, you will be able to work effectively in Databricks.
Requirements:
I am a senior data engineer and trainer, a tech enthusiast, and a father. For more than ten years, I have been passionate about Data Engineering. Initially, I became a self-taught data engineer and then led a team of data engineers at a large company. When I realized the great demand for education in this field, I followed my passion and founded my own Data Engineering Academy. Since then, I have helped over 2,000 students achieve their goals.
| # | Lesson Title | Duration | Access |
|---|---|---|---|
| 1 | Introduction Demo | 02:56 | |
| 2 | Why Databricks | 04:05 | |
| 3 | Pricing explained | 06:51 | |
| 4 | Create Databricks Account & Workspace | 07:09 | |
| 5 | AWS Resources created by Databricks | 04:03 | |
| 6 | Intro Databricks UI & Compute Cluster | 06:05 | |
| 7 | The Dataset | 02:46 | |
| 8 | Goals ETL & Visualization pipeline explained | 02:16 | |
| 9 | Import Data in Databricks UI | 04:44 | |
| 10 | Databricks Data in S3 | 02:09 | |
| 11 | Creating code Repos | 04:35 | |
| 12 | Running our ETL job | 09:26 | |
| 13 | Explore Data Tables in AWS folders | 02:16 | |
| 14 | Explore data with databricks notebook 1 | 05:55 | |
| 15 | Explore data with databricks notebook 2 | 06:45 | |
| 16 | Compute Cluster vs Databricks SQL Warehouse | 04:11 | |
| 17 | Power BI queries through computer cluster | 04:21 | |
| 18 | Power BI queries through Databricks SQL Warehouse | 04:44 | |
| 19 | Conclusion | 02:12 |
Get instant access to all 18 lessons in this course, plus thousands of other premium courses. One subscription, unlimited knowledge.
Learn more about subscription