I have been making a living with web scraping for over 3 years, earned more than $125,000, I have over 25,000 followers on social media, and in this course, I will tell exactly how I scrape websites, as well as how to find clients and grow on social media.
The Ultimate Web Scraping Course
10h 33m 19s
English
Paid
In this course, I will show you how to scrape the following websites (and provide you with the code):
- TikTok
- Zillow
- Google Search
- Google Maps
- Amazon
- Indeed
- ESPN
- Angi List
- And more!
I will also explain how I found clients and gained over 25,000 followers on social media (and how you can do the same).
What's inside the course:
- 9 hours of video materials
- The legality of web scraping: what do you need to know?
- Scrapers vs. hackers: what's the difference?
- All the services I use (proxies, programming languages, servers, databases, packages for bypassing blocks)
- How websites get their data (how APIs work)
- My process for scraping ANY website
- How to find the data source
- How to avoid being blocked
- All about proxies
- Scraping by web frameworks (React, Next.js, static HTML/server-side rendering, ASP.NET, Java)
- How to reverse engineer and bypass websites with good protection
- Scraping websites with cursor-based pagination (e.g., comments on YouTube)
- Scraping Chrome extensions to get free email addresses
- How to scrape mobile apps
- How to search for information about people (finding phone numbers and email addresses)
- How to run Puppeteer on AWS Lambda
- How to intercept requests with Puppeteer
- Solving captchas with Puppeteer
- How to find clients for web scraping
- How to invoice clients
- Product ideas based on web scraping
Disclaimers:
- The code is written in JavaScript, and I use a Mac with VS Code.
- I only briefly mention Puppeteer (there are much BETTER ways).
- Web scraping is not magic. You need a source to scrape from (a website). For example, you cannot scrape people's bank accounts or something similar. That would fall under hacking, which is a different field.
Requirements:
- You should have a good understanding of JavaScript or some other programming language to get the most out of this course.
- If you cannot program at all, this course is not for you.
- I do not teach the basics of programming or JavaScript in this course.
About the Author: Adrian Horning (The Web Scraping Guy)
Adrian Horning publishes under the brand The Web Scraping Guy — focused entirely on the practical side of pulling structured data out of websites that weren't designed to give it up. His material covers the JavaScript and Node.js tooling that dominates modern scraping work (Playwright, Puppeteer, Cheerio, headless browsers, proxy rotation).
The CourseFlix listing carries The Ultimate Web Scraping Course, a paid resource aimed at developers building scraping pipelines — whether for data journalism, market research, or product features that need to consume third-party sites at scale.
Watch Online 53 lessons
0:00
/ #1: 001 The Ultimate Web Scraping Course 2024 - Google Docs - 30 May 2024
All Course Lessons (53)
| # | Lesson Title | Duration | Access |
|---|---|---|---|
| 1 | 001 The Ultimate Web Scraping Course 2024 - Google Docs - 30 May 2024 Demo | 05:27 | |
| 2 | 002 Legality of web scraping | 08:37 | |
| 3 | 003 Scraping vs Hacking | 04:35 | |
| 4 | 004 Services we'll be using in the course | 04:36 | |
| 5 | 005 Services we'll be using in the course 2 | 02:44 | |
| 6 | 006 How websites get their data | 07:58 | |
| 7 | 007 My process for Scraping Any Site | 04:16 | |
| 8 | 008 How to find where the data is coming from | 04:18 | |
| 9 | 009 Proxies | 21:23 | |
| 10 | 010 Scraping Next.js sites (angi, imdb) | 21:17 | |
| 11 | 011 ESPN - Scraping API's | 16:50 | |
| 12 | 012 Amazon - Straight up HTML page | 10:36 | |
| 13 | 013 Update on Overcoming CloudFlare Blocks | 04:23 | |
| 14 | 014 Indeed - Straight Up HTML | 25:14 | |
| 15 | 015 Sarpy - ASP.NET (long version) | 46:08 | |
| 16 | 016 Sarpy - ASP.NET Cliff Notes | 16:15 | |
| 17 | 017 Kitsap - Java Site | 22:07 | |
| 18 | 018 Google Maps | 46:16 | |
| 19 | 019 Linkedin | 08:55 | |
| 20 | 020 How to scrape Shopify | 07:50 | |
| 21 | 021 They can't hide anything from you.... | 01:52 | |
| 22 | 022 Reverse Engineering Realtor API | 27:52 | |
| 23 | 023 Lynchburg exposing Auth Token | 04:03 | |
| 24 | 024 Beacons.ai | 19:45 | |
| 25 | 025 Twitter | 18:29 | |
| 26 | 026 Instagram Scraping Update | 07:38 | |
| 27 | 027 How to Scrape Instagram Profiles | 03:42 | |
| 28 | 028 Instagram | 32:37 | |
| 29 | 029 Arcgis | 16:44 | |
| 30 | 030 Cursor Based Pagination | 11:54 | |
| 31 | 031 Scraping Mobile Apps | 09:34 | |
| 32 | 032 Scraping Zillow Data Tutorial | 11:33 | |
| 33 | 033 Skip Tracing | 09:39 | |
| 34 | 034 Getting Puppeteer Up and Running on AWS Lambda | 15:39 | |
| 35 | 035 facbleag | 08:01 | |
| 36 | 036 Getting Tiktok Videos with Puppeteer | 05:15 | |
| 37 | 037 Solving CAPTCHAs with Puppeteer | 09:44 | |
| 38 | 038 Sitemaps (crunchbase) | 10:29 | |
| 39 | 039 Update 12312024 Use puppeteer to get company pages | 04:59 | |
| 40 | 040 Google Tricks and Tips | 03:49 | |
| 41 | 041 Decoding Imperva Security Measures | 02:49 | |
| 42 | 042 Leveraging Google's Web Cache to get around Cloudflare and Datadome blocking | 03:35 | |
| 43 | 043 Scraping Kick (got-scraping + use your resources) | 08:21 | |
| 44 | 044 I found a way to scrape Ohio State Student Emails | 07:15 | |
| 45 | 045 Tiktok API | 04:35 | |
| 46 | 046 SC Courts, use undetected chrome driver | 02:48 | |
| 47 | 047 Exploring Supabase for Efficient Database Management | 06:32 | |
| 48 | 048 Where to host the scrapers_ Render.com | 04:11 | |
| 49 | 049 Quick note on Concurrency in Web Scraping | 02:44 | |
| 50 | 050 How to make money web scraping | 08:48 | |
| 51 | 051 How to get web scraping clients | 35:52 | |
| 52 | 052 What do you charge clients | 10:04 | |
| 53 | 053 What should you scrape for | 12:42 |
Unlock unlimited learning
Get instant access to all 52 lessons in this course, plus thousands of other premium courses. One subscription, unlimited knowledge.
Learn more about subscriptionRelated courses
-
Updated 1y agoLeetCode & JavaScript Complete Course for Web Developer 2022
By: UdemyIn my LeetCode course, I will walk you through, step-by-step, all the different types of questions that appear during interviews! I am a self-taught programmer2h 53m -
Updated 3y agoShaders for the Web
By: SuperHiLearn how to make OpenGL shaders using the programming language GLSL, and gain insight on how the top websites use hardware-enabled graphics.16h 3m -
ClassicYoutube clone
By: Nomad CodersLearn to create a comprehensive YouTube clone covering front-end and back-end with JavaScript, Node.js, and MongoDB. Enhance your developer skills by building15h 41m5/5
Frequently asked questions
What prerequisites are needed before taking this course?
The course does not explicitly list prerequisites, but a basic understanding of programming and web technologies would be beneficial. Familiarity with JavaScript, as used with Puppeteer, and concepts like HTML, APIs, and proxies may enhance learning, given the course covers topics like scraping HTML pages and using Puppeteer on AWS Lambda.
What kind of projects will I work on during the course?
Throughout the course, you will work on a variety of scraping projects, including scraping data from websites like Amazon, LinkedIn, and Instagram. You'll also learn to scrape Next.js sites and mobile apps, reverse engineer APIs, and employ Puppeteer for tasks like overcoming CAPTCHA challenges and retrieving TikTok videos.
What specific tools and platforms are covered in the course?
The course includes detailed lessons on using Puppeteer, AWS Lambda, and Render.com for web scraping tasks. You will also learn about proxies, using Google's Web Cache, and solving CAPTCHA challenges with Puppeteer. Furthermore, the course discusses leveraging Supabase for database management.
Who is the target audience for this course?
This course is aimed at individuals looking to learn web scraping for personal projects or as a career path. It's also suitable for those interested in finding clients and monetizing web scraping skills, as it includes sections on making money from web scraping and acquiring clients.
How does the scope of this course compare to others?
This course offers a comprehensive approach to web scraping, covering legal aspects, practical scraping techniques, and business-oriented topics like finding clients and growing on social media. Unlike some courses focused solely on the technical side, this one includes lessons on monetization and practical applications.
What topics are not covered in this course?
The course does not cover advanced programming concepts or deep dives into specific programming languages outside of those relevant to web scraping. It also doesn't focus on other data acquisition methods like data mining or focus on ethical hacking beyond differentiating it from scraping.
How much time should I expect to commit to this course?
While the total runtime of the course is not provided, given the 53 lessons, a reasonable estimate would be to allocate several weeks to complete the course if studying part-time. This timeline can vary based on prior knowledge and the time spent on practical exercises and projects.