Skip to main content

Spark Streaming with Scala

11h 17m 52s
English
Paid

Stream big data in real time with Spark and integrate any data source, from Kafka to Twitter.

Nothing static, all in motion.

Your probably know by now: Spark is the most popular computing engine for big data, the most maintained, and with a proven track record of performance. It's 100 times faster than the old MapReduce paradigm, and can easily be extended with machine learning and streaming capabilities, and much more.

In this course, we'll take the natural step forward: process big data as it arrives.

About the Author: rockthejvm.com

rockthejvm.com thumbnail
Rock the JVM aims to be the go-to resource for learning Scala, Akka, Spark and everything JVM. Everything is battle-tested on yours truly. I'm Daniel Ciocîrlan, a developer and teacher from Bucharest, Romania. I've started Rock the JVM to share everything that I learned through pain, blood, sweat and tears and distill everything in action-packed courses with a smooth learning curve. Since 2013, I've been teaching a variety of programming topics to people all over the world and at every level of the game, from senior engineers processing trillions of data points to 7-year-olds moving spaceships on the screen. At first, I mastered Java, then I taught it for 3.5 years to university students. I found Scala 7 years ago when a good friend told me about functional collections. I was blown away, and still am to this day as I discover more of the Scala magic. The rabbit hole only went deeper from there: I learned Akka with my own small projects, then Spark and big data while at Palantir. I left Palantir in mid 2019, but it's very likely my old pipelines are still crunching data every day, as we speak. I hope you enjoy my material - in the meantime, I've just started on Twitter and would love to meet you! A bit more I spend most of my time between Scala, Akka and Spark, and I'm really fortunate I get to work with them and teach them to other developers. I love the expressiveness and the concise power of a good technology. I learn from the community and the people I've had in my courses and I love to share back what I've learned. I love to speak at meetups and conferences and I adore people's faces when something really hard "clicks" in their mind and their face lights up. I don't know how people react to my online courses in real time, but I love the "this changed my life" comments. I've graduated from University Politehnica of Bucharest with a BSc and a MSc in Computer Science. My Bachelor and Master theses were on quantum computation, of all things. That can be explained by the fact that I was really into physics before - I was winning medals at international physics competitions. When I'm not coding, I dance salsa and bachata and I tend to play an absurd amount of Ed Sheeran on my acoustic guitar.

Watch Online 31 lessons

This is a demo lesson (10:00 remaining)

You can watch up to 10 minutes for free. Subscribe to unlock all 31 lessons in this course and access 10,000+ hours of premium content across all courses.

View Pricing
0:00
/
#1: Welcome
All Course Lessons (31)
#Lesson TitleDurationAccess
1
Welcome Demo
18:05
2
Scala Recap
25:06
3
Spark Recap
27:11
4
Spark Streaming First Principles
08:29
5
Streaming DataFrames
27:17
6
Streaming Aggregations
15:58
7
Streaming Joins
25:24
8
Streaming Datasets
26:46
9
Discretized Streams (DStreams)
31:57
10
DStreams Transformations
28:40
11
DStreams Window Functions
32:11
12
Kafka & Structured Streaming
25:08
13
Kafka & DStreams
29:57
14
JDBC with Postgres
11:51
15
Akka and Akka Streams
28:58
16
Cassandra
21:55
17
Setting up a Twitter App
08:17
18
Our First Custom Receiver
15:45
19
Reading Tweets
20:05
20
Reading Tweets: Exercises
15:22
21
Sentiment Analysis on Tweets using NLP
20:50
22
Event Time Windows
31:20
23
Event Time Windows: Exercises
14:54
24
Processing Time Windows
11:16
25
Watermarks
25:16
26
Watermarks, Part 2
25:22
27
Arbitrary Stateful Computation
25:40
28
Arbitrary Stateful Computation
16:36
29
Setting up the REST Server and the Kafka Broker
27:51
30
Integrating Spark Structured Streaming, Test, Run!
33:54
31
You Rock!
00:31
Unlock unlimited learning

Get instant access to all 30 lessons in this course, plus thousands of other premium courses. One subscription, unlimited knowledge.

Learn more about subscription