The Ultimate Web Scraping Course

10h 39m 26s
English
Paid
August 30, 2024

I have been making a living with web scraping for over 3 years, earned more than $125,000, I have over 25,000 followers on social media, and in this course, I will tell exactly how I scrape websites, as well as how to find clients and grow on social media.

More

In this course, I will show you how to scrape the following websites (and provide you with the code):

  • Instagram
  • Twitter
  • TikTok
  • LinkedIn
  • Zillow
  • Google Search
  • Google Maps
  • Amazon
  • Indeed
  • ESPN
  • Angi List
  • And more!

I will also explain how I found clients and gained over 25,000 followers on social media (and how you can do the same).

What's inside the course:

  • 9 hours of video materials
  • The legality of web scraping: what do you need to know?
  • Scrapers vs. hackers: what's the difference?
  • All the services I use (proxies, programming languages, servers, databases, packages for bypassing blocks)
  • How websites get their data (how APIs work)
  • My process for scraping ANY website
  • How to find the data source
  • How to avoid being blocked
  • All about proxies
  • Scraping by web frameworks (React, Next.js, static HTML/server-side rendering, ASP.NET, Java)
  • How to reverse engineer and bypass websites with good protection
  • Scraping websites with cursor-based pagination (e.g., comments on YouTube)
  • Scraping Chrome extensions to get free email addresses
  • How to scrape mobile apps
  • How to search for information about people (finding phone numbers and email addresses)
  • How to run Puppeteer on AWS Lambda
  • How to intercept requests with Puppeteer
  • Solving captchas with Puppeteer
  • How to find clients for web scraping
  • How to invoice clients
  • Product ideas based on web scraping

Disclaimers:

  • The code is written in JavaScript, and I use a Mac with VS Code.
  • I only briefly mention Puppeteer (there are much BETTER ways).
  • Web scraping is not magic. You need a source to scrape from (a website). For example, you cannot scrape people's bank accounts or something similar. That would fall under hacking, which is a different field.

Requirements:

  • You should have a good understanding of JavaScript or some other programming language to get the most out of this course.
  • If you cannot program at all, this course is not for you.
  • I do not teach the basics of programming or JavaScript in this course.

Watch Online The Ultimate Web Scraping Course

Join premium to watch
Go to premium
# Title Duration
1 Is Web Scraping Legal? 08:37
2 Hackers vs Scrapers 04:35
3 My go to packages, editor, language, proxies, etc 04:36
4 How websites get their data/how API’s work 07:58
5 My Process for Scraping Any Site 04:16
6 How to find where data is coming from 04:18
7 How to set up proxies 21:23
8 How to make money web scraping 08:48
9 Scraping Next.js sites (angi, imdb) 21:17
10 Zillow - Scraping Hidden API's 32:31
11 ESPN - Scraping API's 16:50
12 Amazon - Straight up HTML page 10:36
13 Indeed - Straight Up HTML 25:14
14 Sarpy - ASP.NET (long version) 46:08
15 Sarpy - ASP.NET Cliff Notes 16:15
16 Kitsap - Java Site 22:07
17 Google Search 13:37
18 Google Maps 46:16
19 Linkedin 08:55
20 Shopify 07:50
21 They can't hide anything from you..... 01:52
22 Reverse Engineering Realtor API 27:52
23 Lynchburg exposing Auth Token 04:03
24 Beacons.ai 19:45
25 Twitter 18:29
26 Instagram 32:37
27 Arcgis 16:44
28 Cursor Based Pagination (YouTube Comments) 11:54
29 How to scrape Chrome Extensions (free emails) 08:22
30 Reverse Engineering a Private API with mitmproxy 08:25
31 Scraping Mobile Apps 09:34
32 Skip Tracing 09:39
33 Getting Puppeteer Up and Running on AWS Lambda 15:39
34 Getting Tiktok Videos with Puppeteer 05:15
35 Solving CAPTCHAs with Puppeteer 09:44
36 Sitemaps (crunchbase) 10:29
37 Google Tricks 03:49
38 Decoding Imperva Security Measures 02:49
39 Leveraging Google's Web Cache to get around Cloudflare and Datadome 03:35
40 Tiktok API 04:35
41 SC Courts, use undetected chrome driver 02:48
42 Exploring Supabase for Efficient Database Management 06:32
43 Where to host the scrapers? Render.com 04:11
44 Quick note on Concurrency in Web Scraping 02:44
45 How to get web scraping clients 35:52
46 What do you charge clients?? 10:04
47 What should you scrape for $$? 12:42
48 I found a way to scrape Ohio State Student Emails 07:15

Similar courses to The Ultimate Web Scraping Course

Modern JavaScript

Modern JavaScriptui.dev (ex. Tyler McGinnis)

Duration 3 hours 4 minutes 3 seconds
Learn JavaScript Closures (PRO)

Learn JavaScript Closures (PRO)codesmith.io

Duration 5 hours 15 minutes 31 seconds
Build Fullstack Trello clone: WebSocket, Socket IO

Build Fullstack Trello clone: WebSocket, Socket IOudemy

Duration 8 hours 49 minutes 48 seconds