Learning Apache Spark

In this course, you will learn about Spark architecture and the basic principles of its operation. You will practice with transformations and actions in Spark, working in Jupyter Notebooks within a Docker environment. You will also be introduced to DataFrame, SparkSQL, and RDD to understand how to use them for processing structured and unstructured data. Upon completion of the course, you will be ready to write your own Spark jobs and build pipelines based on it.

Basics of Spark

You will understand why using Spark is beneficial, the difference between vertical and horizontal scaling. Learn about the types of data Spark can work with and where it can be deployed.

You will delve into the architecture: executor, driver, context, cluster manager, and the types of clusters. You will also study the differences between client and cluster deployment modes.

Data and Development Environment

You will get acquainted with the tools we will use in the course. You will find out which dataset is chosen and how to set up the working environment: install Docker and Jupyter Notebook.

Spark Coding Basics

Before practice, you will understand the key concepts: what RDD and DataFrame are, and their characteristics when working with different types of data.

You will understand the difference between transformations and actions, how they interact with data and each other, and learn about the most commonly used types of operations.

Practice in Jupyter Notebook

In the GitHub repository, you will find all the code used in the course, allowing you to start working immediately.

You will work with five notebooks where you will learn to:

• apply transformations to data,
• work with schemas, columns, and data types,
• process JSON and CSV in DataFrames,
• merge and modify DataFrames,
• use Spark SQL to work with data,
• apply RDD for unstructured data.