Writing the scraper is the easy part of web scraping. Getting the page is where it breaks.
On our anti-bot test page, a plain HTTP request was turned away in 0.06 seconds. A real Chrome window sat on a Cloudflare check for 30 seconds and never got through.
A scraping API fetched the same page for one credit.
Do you need the paid option? On two of our three test pages, no. A ten-second check in your own browser tells you which one yours needs.
What is web scraping?
Web scraping is using software to load web pages and copy specific data, like prices or reviews, into a spreadsheet, database or file. It’s copy and paste, done on 10,000 pages without getting bored.
Data scraping is the wider term: pulling data from anything a program can read, including PDFs and app screens. Web scraping is the website-only version.
Crawling is different again. A crawler follows links to find pages, the way Google’s bot does. A scraper pulls data from pages you already know, so most jobs crawl a shop’s categories first, then scrape each product.
Scraping also overlaps with browser automation when data only appears inside a browser. Our browser automation vs web scraping guide draws that line.
How web scraping works, from request to spreadsheet
Every scraper does the same four jobs: fetch the page, find the data, clean it and save it.
- Fetch. The scraper sends the same HTTP request your browser sends and gets HTML back. On a page that builds itself with JavaScript, that HTML is an empty shell.
- Parse. A parser turns the HTML into a tree. The scraper picks elements with CSS selectors such as
p.price_color, which reads as “the price under every book”. - Clean. Fix the text encoding, turn “£51.77” into 51.77 and drop duplicates.
- Store. Write rows to a CSV, a Google Sheet, a database or an AI pipeline.
Here are the first two steps for our static test page, in 11 lines of Python with the requests and BeautifulSoup libraries:
import requests
from bs4 import BeautifulSoup
page = requests.get("https://books.toscrape.com/", headers={"User-Agent": "Mozilla/5.0"})
page.encoding = "utf-8" # the server doesn't say, so requests guesses wrong
soup = BeautifulSoup(page.text, "html.parser")
for book in soup.select("article.product_pod"):
title = book.select_one("h3 a")["title"]
price = book.select_one("p.price_color").text
print(title, price)
Line 5 exists because our first run printed “£51.77” for every price. The server never names its character set, so requests guessed wrong. Nothing crashed, which is how most scraping bugs look.

Schedule that script daily and you’ve built a price tracker. Point it at 10,000 URLs on a site that doesn’t want you, and you’ll meet every problem below.
What is web scraping used for?
Mostly for watching things that change:
- Price monitoring. A shop checks 300 competitor product pages every morning and reprices when a rival drops below it.
- SEO rank tracking. Pulling the top 10 results for your keywords. Google fights this harder than almost anyone, and our guide to scraping Google search results safely covers what still works.
- Lead lists. Company names, job titles and public contact pages. LinkedIn is the riskiest target, so read how to scrape LinkedIn data safely first.
- AI and research. Clean page text for a chatbot, or 5,000 reviews mined for what customers complain about.
Is your job one page you check once a week? Don’t build anything. Bookmark it.
Three ways to scrape the same page: how we tested
Short version: three practice pages built for scrapers, three methods, and no real company’s site.
- A static bookshop page (Books to Scrape) with 20 books in plain HTML.
- A JavaScript product grid (ScrapingCourse’s rendering challenge) whose 12 products appear only after scripts run.
- An anti-bot page (ScrapingCourse’s antibot challenge) behind a Cloudflare check.
Each page got five runs with plain HTTP, using Python’s requests plus BeautifulSoup. It got five more with a headless browser: Playwright driving a fresh Chromium with no visible window. The API leg was one Firecrawl call per page through its MCP connector.
Everything ran from one Linux server with a datacenter IP and no proxy. We didn’t time Firecrawl, because the connector adds its own delay. We didn’t run Apify, ScraperAPI or a proxy network, so their costs below come from price lists, not our bills.
The Firecrawl, Apify, ScraperAPI, Bright Data and Decodo links on this page are affiliate links. We may earn a commission at no cost to you.
Same three pages, three methods: who got the data?
Plain HTTP was enough for the static page. The JavaScript page needed a browser. Only the API got past the anti-bot check.
| Test page | Plain HTTP | Headless browser | Firecrawl API |
|---|---|---|---|
| Static books page | 20 of 20 books, 2.25 s | 20 of 20 books, 2.72 s | 20 of 20 books, 1 credit |
| JavaScript product grid | 0 of 12 (empty cards) | 12 of 12, 0.49 s | 12 of 12, 1 credit |
| Cloudflare anti-bot page | Blocked (403) in 0.06 s | Blocked at the human check | Got through, 1 credit |
The static page: a browser was dead weight
The plain request pulled all 20 titles and prices in a median 2.25 seconds. Most of that was the sandbox’s slow server, not our code.
Headless Chromium got the same 20 in 2.72 seconds, loading 29 or 30 images, styles and scripts along the way.
Memory is the bigger gap. Chromium’s processes held about 600 MB by our count, against 42 MB for the Python script. Twenty scrapers side by side means roughly 12 GB of RAM versus under 1 GB.
The JavaScript page: a 200 status and nothing in it
The raw HTML held 12 product cards with nothing inside them. No names, no prices.
That’s the trap. The request reported success, status 200, with zero products. Nothing errors, so a scheduled job saves empty rows for weeks.
Playwright waited for the names to appear and read all 12 in a median 0.49 seconds. Firecrawl returned all 12 as Markdown with prices and links, plus the page’s stray “Enable JavaScript to see products” notice.
Real-world tip: before you reach for a browser, open DevTools, click the Network tab and reload. Many JavaScript pages fetch their data as JSON from a separate address, and calling that directly beats both methods.
The anti-bot page: the browser didn’t help
Plain requests got a 403 error and a “Just a moment…” page on all five runs, in 0.06 seconds. Cloudflare decided before sending a byte of the real page.
Headless Chromium got the same 403 and a “Verify you are human” box. So we gave a visible Chromium window, then Google Chrome itself, 30 seconds each. Neither got past the check. We didn’t click the box or use a CAPTCHA solver.
Firecrawl’s first call returned the page’s success message, “You bypassed the Antibot challenge!”, for one credit on its basic proxy setting.
A real browser fixed the JavaScript problem. It didn’t change who we looked like.
Why do websites block scrapers, and what gets past them?
Sites block scrapers by judging who’s asking. Four signals do most of the work:
- The IP address. Datacenter IPs like our server’s are easy to flag. Home and mobile IPs look like customers. Supplying them is what proxies are for, and our explainer on the role of proxies in web scraping covers the types.
- The request’s fingerprint. A Python request wearing a Chrome user-agent string still sends different headers and a different TLS handshake from real Chrome.
- JavaScript challenges. The check runs code in your browser and scores the result. Automated browsers leave traces these tests look for, the likeliest reason ours failed.
- Speed. A hundred requests a minute from one address isn’t a person shopping.

Three fixes get you past them, cheapest last:
- Better IPs. A residential proxy network such as Bright Data or Decodo rotates your requests through home IPs. You still write and run the scraper.
- A scraping API. ScraperAPI and Firecrawl handle proxies, browsers and retries for you, and charge per page.
- Going slower. Fewer requests, spread out, with pages cached so you never fetch one twice. It’s free, and most people skip it.
What does web scraping cost?
Plain HTTP and a headless browser cost nothing per page; you pay for the server and, on protected sites, proxies. Scraping APIs charge per page, more on hard sites.
We checked these prices on 21 November 2026:
- Firecrawl: 1,000 free credits a month, one credit per normal page. Hobby is $19 for 5,000 credits, or $3.80 per 1,000 pages. Standard is $99 for 100,000, or $0.99 per 1,000. A page that comes back as a 403 or 404 still costs a credit.
- ScraperAPI: Hobby is $49 for 100,000 credits, or $0.49 per 1,000 ordinary pages. Cloudflare-protected sites add 10 credits a request, so 1,000 of those cost about $5.39. LinkedIn pages cost 30 credits each, or $14.70 per 1,000.
- Do it yourself: no per-page fee. Each browser took around 600 MB in our test, though, and residential proxies are billed on top, usually per GB.
See the pattern? The price list quotes a page. Your bill counts attempts, including the failures and every protected domain.
Our Firecrawl review works through its credit maths on real jobs. Our Apify review covers renting a ready-made scraper instead.
Where web scraping hits its limits
A scraper can read anything a page shows the public. That doesn’t mean you may take it all, or that the scraper keeps working.
- Logins. Scraping behind a login means you’ve accepted the site’s terms, and breaking them gets accounts banned.
- Personal data. Names, emails and profiles bring privacy law such as GDPR into play, even on public pages. Our plain-English guide to whether web scraping is legal shows where the lines sit.
- Breakage. Our script depends on the class name
product_pod. A redesign that renames it leaves you with zero rows, so alert on empty runs. - Messy output. The bookshop’s link text cuts long titles to “A Light in the …”, with the full title hidden in an attribute. Check a sample row by eye, whatever you use.
- Scale. A few hundred pages a day is easy. Millions need queues, retries and storage, which is where our best web scraping tools list starts paying off.
What about robots.txt? It’s the file where a site lists the paths it wants bots to skip. It isn’t a law, but ignoring it looks bad the day the site complains.
Which scraping method should you start with?
Here’s the ten-second check. Open the page, press Ctrl+U to view its source, and search for a price or name you can see on screen.
- It’s in the source: use plain HTTP, because it’s the cheapest to run and the easiest to debug. Best for static shops, news sites and docs.
- It’s missing from the source: use a headless browser like Playwright, or hunt for the JSON request first. Best for developers scraping JavaScript apps they’re allowed to scrape.
- You get a 403, a “Just a moment” page or a CAPTCHA: use a scraping API or a proxy network. Best for small teams who’d rather pay per page than babysit proxies.
- You don’t code at all: start with Apify‘s ready-made scrapers, because someone else fixes the selectors when the site changes.
Our take: start with plain HTTP every time. It failed on two of our three pages, but one request tells you whether you need more. When it works, nothing is cheaper.
Frequently asked questions
Is web scraping the same as using an API?
No. An official API is a door the site built for programs, with documented fields and published limits. Scraping reads pages built for people. If a site offers an API with the data you need, use it. It’s steadier, and its terms usually allow it. Scrape only what the API leaves out.
Can I do web scraping without coding?
Yes. No-code tools and browser extensions let you click the data you want and export a CSV. Apify’s ready-made Actors cover common targets like Google Maps and online shops. You won’t write Python, but you’ll still need to check the output and fix things when a site changes its layout.
What is web scraping likely to cost for 10,000 pages a month?
Under $50 on ordinary pages. Firecrawl’s Hobby plan covers 5,000 pages for $19, and top-ups add 1,000 credits per $5, so 10,000 pages comes to about $44. ScraperAPI’s $49 Hobby plan covers them too, unless they sit behind Cloudflare. At 11 credits each, 100,000 credits stretch to about 9,090 pages.
How do I stop my scraper from getting blocked?
Slow down before you spend money. Add random pauses of a few seconds between requests and reuse one session so cookies carry over. Cache every page so you never fetch it twice, and scrape at quiet hours. Still blocked after that? Switch to residential proxies or a scraping API.
Can websites tell I’m scraping them?
Usually, yes, once you send more than a handful of requests. Server logs show your IP address, your timing and your headers, and bot-protection services compare those against what real visitors look like. A few polite requests rarely get noticed. Thousands from one datacenter IP almost always do.
What’s the best programming language for web scraping?
Python, for most people. Requests, BeautifulSoup, Scrapy and Playwright all have mature Python versions, and most tutorials use it. JavaScript on Node.js is the other strong pick, especially with Puppeteer or Playwright, if your team already writes it. The language matters far less than how you handle blocks.
Do I need proxies for web scraping?
Not for small jobs on unprotected sites. You need them when a site rate-limits or bans your IP, or shows different prices and results by country. Heavy volume is the other trigger. Start with a few residential IPs, and add more only if the blocks continue.
The bottom line
Web scraping is four steps: fetch, parse, clean, store. The method only changes the first one, and that’s the step that breaks.
Data in the page source? Plain HTTP gets it free. A page JavaScript builds needs a headless browser. A bot check needs a scraping API, at a per-page price that rises on protected sites.
Before you scale past a test, check the site’s terms and whether you’re collecting personal data. Then run the ten-second check on your page and try the 11-line script on it.



