How to Scrape Websites for LLMs 2026: Turning Pages into Clean Markdown

Your RAG bot just quoted a cookie banner back to a customer.

Cleaner input fixes that. To scrape websites for LLMs, take the site’s own Markdown when it has one. Otherwise run the HTML through trafilatura, check what survived, and chunk by heading.

On eight real pages, our RAG explainer shrank from 119,965 tokens of HTML to 2,921 of clean Markdown.

The catch: even the best extractor lost content on half the pages.

Firecrawl, Apify and n8n links are affiliate links. They never changed a result.

What does LLM-ready Markdown actually look like?

LLM-ready Markdown is the page’s main content and nothing else, with its headings, tables and code still intact. The URL and title travel with it as metadata.

Four things decide whether a scraped page helps your model or quietly hurts it.

• Main content only. No menus, footers, cookie notices or “related posts” lists.
• Structure kept. Headings, tables and code fences survive, because your chunker and your model both lean on them.
• Few inline links. A URL inside a sentence costs tokens and adds nothing to an embedding.
• Metadata on the side. Page URL, title, section and fetch date, so answers can cite a source.

Why fuss? Noise isn’t harmless. Converted whole, our plain-English RAG explainer came to 6,933 tokens. “View All Result”, “Subscribe” and the privacy-policy footer were all in there.

The clean version was 2,921 tokens. You’d embed, store and search the other 4,000 for nothing.

How we tested this (and what we didn’t)

On 30 October 2026 we fetched six pages: two DroidCrunch posts, Wikipedia’s web scraping article, a Python docs page, MDN’s Fetch guide and a GitHub README. Stripe’s and Firecrawl’s docs joined for Step 1.

Each page went through five open-source converters and Jina’s free Reader. We counted tokens with OpenAI’s o200k tokenizer. Then we checked that five passages per page survived, and that the menus didn’t.

Next we chunked the output two ways, embedded it with the open-source bge-small model, and asked 14 questions with known answers. It’s a small test. Read the retrieval numbers as a direction, not a benchmark.

Not tested: Firecrawl with a paid key, pages that only render in a browser, and PDFs.

Step 1: does the site already give you Markdown?

More docs sites do than you’d guess. Try the page URL with .md on the end, then check for /llms.txt at the root.

Stripe’s payments overview shows why this comes first. The HTML weighed 233,625 tokens. Trafilatura squeezed it to 81, and that’s the problem: it kept the headings and threw away nearly everything under them.

The site’s own payments.md was 1,184 tokens, and every word was there.

An llms.txt file is a Markdown index of a site’s key pages, written for AI tools. Ours lists 63 posts with a one-line summary each. That’s a ready-made crawl list.

No Markdown version? Most sites won’t have one. Fetch the HTML.

Step 2: fetch the HTML politely, and dodge the robots.txt trap

Send a user agent that names your bot, honour robots.txt, and keep the rate low. This script handles the first two and tries the .md version before the HTML. The loop in Step 6 adds a two-second pause between pages.

import requests
from urllib.parse import urlparse
from urllib.robotparser import RobotFileParser

UA = "MyRAGBot/1.0 (+https://example.com/bot)"

def allowed(url):
    root = "{0.scheme}://{0.netloc}".format(urlparse(url))
    # rp.read() would fetch as "Python-urllib", which many sites answer with 403
    r = requests.get(root + "/robots.txt", headers={"User-Agent": UA}, timeout=20)
    rp = RobotFileParser()
    rp.parse(r.text.splitlines() if r.ok else [])
    return rp.can_fetch(UA, url)

def fetch(url):
    if not allowed(url):
        raise PermissionError(f"robots.txt blocks {url}")
    # 1. Does the site already publish Markdown?
    md = requests.get(url.rstrip("/") + ".md", headers={"User-Agent": UA}, timeout=20)
    if md.ok and "markdown" in md.headers.get("content-type", ""):
        return "markdown", md.text
    # 2. Otherwise take the HTML
    r = requests.get(url, headers={"User-Agent": UA}, timeout=20)
    r.raise_for_status()
    if "charset" not in r.headers.get("content-type", "").lower():
        r.encoding = r.apparent_encoding   # stops "—" turning into "—"
    return "html", r.text

Our first version read robots.txt the textbook way, with RobotFileParser.read(). It blocked our own blog.

Why? That method fetches robots.txt as Python-urllib. DroidCrunch and Wikipedia both answer that user agent with a 403, and Python treats a 403 as “everything is disallowed”. Fetching the file yourself with your real user agent fixes it.

Common mistake: trusting r.text on pages that don’t declare a charset. Python’s docs send a bare text/html header, so requests guessed the wrong encoding and every dash in the title turned into —. The apparent_encoding line stops that.

Scraping more than a few hundred pages from one site? Slow down first. If you still get blocked, our guide to rotating proxies in Python covers the next step. And before you point a crawler at someone else’s content, read whether web scraping is legal for your use.

Step 3: which extractor gives you the cleanest Markdown?

Trafilatura, with a simple fallback for the pages it mangles. That pairing kept every test passage on all six HTML pages.

Here’s what each converter produced, in tokens. Fewer is better only when nothing important went missing.

✗ marks output that lost content we checked for. “Whole page as Markdown” is html2text run on the full page. Markdownify and Microsoft’s MarkItDown came within 2% of it across the six pages, menus included.

So the whole-page converters are out. They keep everything, junk included. The real contest is between the tools that hunt for the article.

Where each extractor broke

• Readability (the readability-lxml port) kept one section of the Wikipedia article and dropped the rest: 1 of our 5 test passages. It also stripped every heading from MDN’s guide.
• Trafilatura flattened the GitHub README to zero headings and skipped one method on the Python docs page. On Firecrawl’s docs it dropped all 17 code samples.
• Jina Reader returned MDN’s guide with none of its 23 code samples. With the X-Engine: direct header and caching off, 22 came back. GitHub refused it outright: someone else’s traffic had got anonymous access to github.com blocked.
• The main-content selector never lost a test passage. It did keep MDN’s “Was this page helpful?” footer, because that sits inside the article.

Trafilatura still gets the default slot, because its misses are easy to detect and its output is the tightest. The selector is the safety net.

import re
import trafilatura
from bs4 import BeautifulSoup
from markdownify import markdownify

DROP = "nav, header, footer, aside, script, style, form, svg, [role=navigation]"

def main_content(html):
    soup = BeautifulSoup(html, "lxml")
    root = soup.select_one("main, article, [role=main], #content") or soup.body
    for tag in root.select(DROP):
        tag.decompose()
    return markdownify(str(root), heading_style="ATX", strip=["a", "img"],
                       escape_underscores=False, escape_asterisks=False)

def looks_broken(md, html):
    code_in_html = html.count("<pre")
    code_in_md = md.count("```") // 2
    return (len(md) < 500                         # almost nothing came out
            or (code_in_html and code_in_md == 0)  # code samples vanished
            or md.count("\n#") < 2)                # headings vanished

def to_markdown(html):
    md = trafilatura.extract(html, output_format="markdown",
                             include_tables=True, include_formatting=True) or ""
    used = "trafilatura"
    if looks_broken(md, html):
        md, used = main_content(html), "fallback"
    title = BeautifulSoup(html, "lxml").title
    title = title.get_text(strip=True) if title else ""
    return title, re.sub(r"\n{3,}", "\n\n", md).strip(), used

Speed doesn’t decide it. Every local extractor finished a page in under a tenth of a second. Jina’s hosted reader took 1.5 to 2.6 seconds, since it fetches and renders the page itself.

Step 4: check the Markdown before it reaches your vector store

Three cheap checks catch most broken extractions: almost no text, code blocks that vanished, or headings that vanished. That’s the looks_broken() function above.

On our run it flagged the GitHub README and the Python docs page, then swapped in the selector output. Both came back complete, skipped method included.

It flags trafilatura’s Stripe and Firecrawl output too. We never needed it there, because Step 1 had already found their Markdown.

Then come the quiet errors that no size check will catch.

• Escaped underscores. Markdownify’s default turns site_maps() into site\_maps(). A search for the function name then misses. Set escape_underscores=False.
• Hidden emails. Cloudflare’s email protection meant the support address in our own Firecrawl FAQ came out as “[email protected]” in every extractor.
• Wiki leftovers. Wikipedia text arrives with [edit] links and citation numbers like [11]. They’re harmless, but a regex pass trims them.

Real-world tip: open three converted pages from each site template and read them top to bottom. That’s how we spotted the escaped underscores and the hidden email. No check flagged either one.

Clean pages are half the job. How you cut them up decides what the model actually sees.

Step 5: chunk by heading, not by token count

Heading-aware chunks put the right answer first more often, sent less text to the model and never broke a code block.

We cut the same six pages two ways. Fixed chunks are 300-token windows with a 30-token overlap. Heading-aware chunks pack whole paragraphs under each heading, up to 300 tokens, with the page title and heading path on top.

Cleaning did the first job. The whole-page corpus was 54,372 tokens against 23,849 clean, and it missed two questions entirely.

Chunking by heading did the second. It sent 37% fewer tokens per answer. Only 4 of its 149 chunks opened on a lowercase word, the mark of a mid-sentence cut, against 43 of 82 fixed windows.

import re
import tiktoken

enc = tiktoken.get_encoding("o200k_base")
tokens = lambda s: len(enc.encode(s, disallowed_special=()))

def blocks(md):
    # split on blank lines, but never inside a ``` code fence
    out, cur, fence = [], [], False
    for line in md.split("\n"):
        if line.strip().startswith("```"):
            fence = not fence
        if not fence and not line.strip() and cur:
            out.append("\n".join(cur)); cur = []
        elif line.strip() or fence:
            cur.append(line)
    return out + (["\n".join(cur)] if cur else [])

def split_long(text, max_tokens):
    # break an oversized paragraph or list at sentence or line ends
    parts, cur = [], ""
    for s in re.split(r"(?<=[.!?])\s+|\n", text):
        if cur and tokens(cur + " " + s) > max_tokens:
            parts.append(cur); cur = s
        else:
            cur = (cur + " " + s).strip()
    return parts + ([cur] if cur else [])

def chunk(md, title, url, max_tokens=300):
    path, buf, chunks = {}, [], []
    def flush():
        if buf:
            crumb = " > ".join([title] + [path[k] for k in sorted(path)])
            chunks.append({"url": url, "section": crumb,
                           "text": crumb + "\n\n" + "\n\n".join(buf)})
            buf.clear()
    for b in blocks(md):
        h = re.match(r"^(#{1,6})\s+(.+)$", b)
        if h:
            flush()
            level = len(h.group(1))
            path = {k: v for k, v in path.items() if k < level}
            path[level] = h.group(2).strip()
        else:
            pieces = [b]
            if tokens(b) > max_tokens and not b.startswith("```"):
                pieces = split_long(b, max_tokens)   # code blocks stay whole
            for p in pieces:
                if buf and tokens("\n\n".join(buf + [p])) > max_tokens:
                    flush()
                buf.append(p)
    flush()
    return chunks

Why 300 tokens? The bge-small model cuts off anything past 512 of its own tokens, so 300 by OpenAI’s count leaves headroom. With a bigger embedding model, go to 500 or so, then rerun your test questions.

Look at what the breadcrumb does. A Wikipedia chunk opens with Web scraping - Wikipedia > Web scraping > Legal issues. Even a short chunk now tells the embedding what it’s about.

Step 6: store each chunk with the metadata your RAG app needs

Every chunk should carry its URL, heading path, fetch date and a content hash. The URL lets the model cite its source. The hash lets you skip re-embedding pages that haven’t changed.

import json, hashlib, time
from fetch_page import fetch
from to_markdown import to_markdown
from chunk_markdown import chunk

def ingest(url, out="chunks.jsonl"):
    kind, body = fetch(url)
    if kind == "markdown":
        title, md, used = url, body, "native .md"
    else:
        title, md, used = to_markdown(body)
    with open(out, "a") as f:
        for c in chunk(md, title or url, url):
            c["fetched_at"] = time.strftime("%Y-%m-%d")
            c["hash"] = hashlib.sha256(c["text"].encode()).hexdigest()[:16]
            f.write(json.dumps(c, ensure_ascii=False) + "\n")
    return used

for url in open("urls.txt").read().split():
    print(url, ingest(url))
    time.sleep(2)   # be polite

Our eight URLs produced 170 chunks and 26,096 tokens in 11.9 seconds. Two pages came from native Markdown, two from the fallback and four straight from trafilatura.

That JSONL file loads into any vector store. Rather not write the retrieval side? Our RAG chatbot tutorial in n8n builds it without code, and n8n self-hosts for free.

On a re-crawl, hash each new chunk and compare. Only the changed ones need new embeddings, which matters once you’re past a few thousand pages.

When is a hosted scraper worth paying for?

Pay when pages need a real browser, when the site fights bots, or when a crawl runs to thousands of pages. For a docs site that serves plain HTML, the scripts above cost nothing.

Firecrawl is the one we’d pick for an AI app, because one API call returns Markdown and the credit maths is simple. Our Firecrawl pricing breakdown shows which plan fits your volume, and the Firecrawl tutorial walks through your first scrape and crawl.

Apify fits when you want a whole site crawled on a schedule without writing code. Its crawler uses Mozilla’s Readability by default, so if pages come back thin, switch the HTML transformer to Defuddle and compare. Our guide to Apify Actors shows how runs and datasets work.

Torn between Firecrawl and Apify? Our Firecrawl vs Cloro vs Apify comparison weighs them, plus a third option, on cost and output.

Our take: start local. A hosted tool doesn’t excuse you from Step 4. Jina’s missing code samples looked fine until we counted them.

Frequently asked questions

Which setup should you use to scrape websites for LLMs?

When you scrape websites for LLMs, ask for the site’s own Markdown first. Nothing you build will beat the author’s version.

For everything else, run trafilatura, keep the selector fallback, and run the three checks on every page. Then chunk by heading, so each piece of clean Markdown carries its own context into your RAG pipeline.

Reach for Firecrawl or Apify when pages need a browser or the crawl gets big.

One warning: no extractor is safe unchecked. Trafilatura silently lost code, headings or whole sections on half our pages.

Your next step: run pipeline.py on ten of your own URLs and read three of the outputs before you embed anything.