CF

Data Engineering with Hadoop

7h 3m
English
Paid

Big Data is not just a buzzword, but a real phenomenon. Every day, companies around the world collect and process vast amounts of data at high speeds. This data is often unstructured and inconsistent, making it nearly impossible to process using traditional methods. One of the platforms that has proven itself for working with big data is Apache Hadoop. This is an open-source framework in Java that allows processing and storing large volumes of data in clusters using simple programming models. Hadoop is a flexible, fast, and affordable architecture capable of detecting and handling failures at the application level.

Course Overview

In this course led by Suyog Nagaokar, you will gain a comprehensive understanding of the Hadoop architecture and its components:

  • HDFS (Hadoop Distributed File System) - for large-scale storage
  • YARN (Yet Another Resource Negotiator) - for resource management
  • MapReduce - for data processing
  • Hive - for SQL-like querying
  • Sqoop - for transferring data between Hadoop and relational databases

The course includes both theoretical foundations and practical lab exercises. By the end of the course, you will be able to:

  • Grasp the essential elements of the Hadoop ecosystem
  • Execute basic Hadoop commands
  • Create solutions using each Hadoop component for tackling real-world business challenges

Practical Application

You will install and configure a full Hadoop environment using the Cloudera Quickstart VM directly on your computer. In practice, you will learn to:

  • Utilize Sqoop, Hive, and MySQL for data storage and querying
  • Craft and execute Hive queries for data analysis on Hadoop
  • Manage data clusters efficiently using HDFS, MapReduce, and YARN
  • Operate clusters with the Hue interface

Course Requirements

  • A PC with a 64-bit version of Windows or Linux and internet access
  • At least 8 GB of free (not total) RAM to complete practical tasks (having less will allow you to follow along with the training theory but without practice)
  • Basic programming skills, preferably with Python
  • Familiarity with the Linux command line is highly advantageous

This course is ideal for both beginners and those who wish to expand their knowledge in Big Data and master one of the industry's most popular frameworks.

Additional

https://github.com/team-data-science/Hadoop-Suyog-Nagaokar

About the Author: Suyog Nagaokar

Suyog Nagaokar thumbnail

Suyog Nagaokar is a software engineer and educator focused on the Hadoop / big-data ecosystem — the foundational platform for processing data at scale that anchored a generation of data-engineering work.

His CourseFlix listing carries Data Engineering with Hadoop — a structured treatment of the Hadoop ecosystem: HDFS, MapReduce, YARN, the Hive / Pig / Spark layers on top, and the operational patterns for running Hadoop clusters in production.

Material is paid and aimed at data engineers picking up Hadoop for legacy or current production systems. For broader content, see CourseFlix's Data processing and analysis category page.

Watch Online 45 lessons

This is a demo lesson (10:00 remaining)

You can watch up to 10 minutes for free. Subscribe to unlock all 45 lessons in this course and access 10,000+ hours of premium content across all courses.

View Pricing
0:00
/
#1: What can you expect from this course?
All Course Lessons (45)
#Lesson TitleDurationAccess
1
What can you expect from this course? Demo
02:10
2
Introduction to Big Data
14:50
3
What is Hadoop? Why Hadoop?
05:38
4
Hadoop Architecture – Overview
02:39
5
Hadoop Architecture – Key services
07:13
6
Storage/Processing characteristics
07:51
7
Store and process data in HDFS
03:56
8
Handling failures - Part 1
05:10
9
Handling failures - Part 2
07:33
10
Rack Awareness
05:59
11
Hadoop 1 v/s Hadoop 2
12:51
12
Hadoop Ecosystem
03:36
13
Vanilla/HDP/CDH/Cloud distributions
10:12
14
Install Cloudera Quickstart Docker
07:19
15
Hands-on with Linux and Hadoop commands
05:49
16
Hive Overview
04:54
17
How Hive works
05:57
18
Hive query execution flow
04:59
19
Creating a Data Warehouse & Loading data
05:10
20
Creating a Hive Table
21:19
21
Load data from local & HDFS
17:19
22
Internal tables vs External tables
17:20
23
Partitioning & Bucketing. (Cardinality concept)
16:24
24
Static Partitioning - Lab
14:58
25
Dynamic Partitioning - Lab
13:55
26
Bucketting - Lab
22:32
27
Storing Hive query output
11:34
28
Hive SerDe
14:26
29
ORC File Format
14:10
30
Sqoop overview
03:52
31
Sqoop list-databases and list-tables
06:31
32
Scoop Eval?
03:59
33
Import RDBMS table with Sqoop
11:40
34
Handling parallelism in Sqoop
09:02
35
Import table without primary key
11:01
36
Custom Query for Sqoop Import
08:48
37
Incremental Sqoop Import - Append
09:52
38
Incremental Sqoop Import - Last Modified
13:55
39
Scoop Job
08:01
40
Sqoop Import to a Hive table
10:59
41
Sqoop Import all tables - Part 1
06:20
42
Sqoop Import all tables - Part 2
14:03
43
Sqoop Export
06:14
44
Export Hive table
04:36
45
Export with Staging table
06:24
Unlock unlimited learning

Get instant access to all 44 lessons in this course, plus thousands of other premium courses. One subscription, unlimited knowledge.

Learn more about subscription

Related courses

Frequently asked questions

What are the prerequisites for enrolling in this course?
The course does not explicitly list prerequisites, but a basic understanding of Java programming and familiarity with Linux commands may be beneficial. The course includes hands-on exercises with Linux and Hadoop commands, suggesting prior exposure to these areas could enhance the learning experience.
What kind of projects will I work on during the course?
Students will install and configure a full Hadoop environment using the Cloudera Quickstart VM, which involves practical exercises such as creating Hive tables, loading data, and using Sqoop for data transfer between Hadoop and relational databases. These projects are designed to tackle real-world business challenges using Hadoop components.
Who is the target audience for this course?
The course is suitable for individuals interested in building a career in data engineering, particularly those who want to gain practical experience with Hadoop. It is likely beneficial for software developers, data analysts, and IT professionals looking to expand their knowledge in big data technologies.
How does this course compare in depth to other data engineering courses?
This course provides a comprehensive understanding of the Hadoop ecosystem, including its architecture and components like HDFS, YARN, MapReduce, Hive, and Sqoop. It includes both theoretical foundations and practical lab exercises, making it suitable for those seeking an in-depth exploration of Hadoop specifically, compared to broader data engineering courses.
What specific tools or platforms will I learn to use in this course?
The course covers key tools and platforms within the Hadoop ecosystem, including HDFS for storage, YARN for resource management, MapReduce for data processing, Hive for SQL-like querying, and Sqoop for data transfer between Hadoop and relational databases. Students will also use the Cloudera Quickstart VM for practical exercises.
What topics or tools are not covered in this course?
While the course covers several essential components of the Hadoop ecosystem, it does not delve into other big data technologies or platforms such as Apache Spark, Kafka, or NoSQL databases. The focus is specifically on Hadoop and its core components.
How much time should I expect to commit to this course?
The course consists of 45 lessons that include both theoretical content and practical lab exercises. While the exact runtime is not specified, students should be prepared to dedicate time to both learning the material and completing hands-on projects to fully grasp the concepts and applications of Hadoop.