---
source_url: "https://alterlab.io/blog/web-scraping-pipeline-for-rag-clean-data-for-llms"
title: "Web Scraping Pipeline for RAG: Clean Data for LLMs | AlterLab"
mirrored_at: 2026-08-11T03:33:49.537Z
host: alterlab.io
cited_in_42a: true
mirror_canonical: "https://index.42a.ai/alterlab.io/blog/web-scraping-pipeline-for-rag-clean-data-for-llms"
---

> **Original source:** https://alterlab.io/blog/web-scraping-pipeline-for-rag-clean-data-for-llms

AlterLab handles this automatically — scrape any URL with one API call. No infrastructure required.

[Try it free](https://alterlab.io/playground)

Raw HTML is poison for RAG. A typical news article page is 45,000 characters—roughly 11,000 tokens. The actual article is 800 words, or about 1,100 tokens. You are paying 10× to embed navigation menus, cookie banners, footer links, and inline scripts that actively dilute your embeddings and degrade retrieval quality.

The fix is a five-stage pipeline: reliable fetch → content extraction → normalization → semantic chunking → embed and index. Each stage has a single responsibility. Each failure is isolated and debuggable. This post walks through a production implementation in Python.

* * *

## Pipeline Architecture

* * *

## Stage 1: Reliable Fetching

The hardest part of scraping at scale is not parsing—it is getting the HTML. Bot detection blocks `requests`. JavaScript-rendered SPAs return skeleton HTML to static fetches. IP ranges accumulate blocks.

AlterLab's scraping API handles this in a single POST: rotating residential proxies, automatic CAPTCHA bypass, and optional headless rendering without managing a browser fleet yourself.

**Python:**

**cURL:**

* * *

## Stage 2: Content Extraction

`trafilatura` is the most accurate open-source library for pulling article body text from HTML. It outperforms `readability-lxml` and `newspaper3k` on structured documentation and blog content because it uses both DOM heuristics and text-density scoring.

Set `no_fallback=False` to allow trafilatura to fall back to its secondary heuristic if the primary DOM analysis returns nothing—useful for pages with unconventional layouts.

* * *

## Stage 3: Normalization

After extraction, text still contains artifacts: Unicode non-breaking spaces (`\u00a0`), zero-width joiners, smart quotes, triple-newline runs from CMS templates, and stub lines that are purely punctuation.

This pass runs in microseconds per document and prevents garbage tokens from reaching your embedding model.

* * *

## Stage 4: Chunking Strategy

Three mistakes that kill retrieval quality:

-   **Fixed character splits** break sentences mid-clause. The embedding for a sentence fragment does not represent a complete thought.
-   **Whole documents as single vectors** average all content into one point in embedding space. Specific queries retrieve nothing useful.
-   **Zero overlap** means a concept bridging two chunks never matches a query that references it as a unit.

Use recursive sentence-aware chunking with configurable overlap:

**Token ceiling guidelines by model:**

* * *

## Stage 5: Embedding and Indexing

Batch your embedding calls. The OpenAI embeddings API accepts up to 2,048 inputs per request—sending one chunk per call is 100× slower and burns rate limit quota unnecessarily.

Store `text` in the vector metadata. Fetching the source document at query time adds latency and a failure point; paying a few extra bytes per vector is worth it.

* * *

## Full Pipeline

* * *

## Handling Edge Cases

### Deduplication

The same content appears under multiple URLs: `www` vs. bare domain, query parameters, pagination suffixes. Hash normalized text before indexing:

Call `is_duplicate(clean_text)` after Stage 3 and skip to the next URL if it returns `True`.

### Pagination and Crawling

For documentation sites spanning dozens of pages, discover internal links before ingesting. A simple same-domain BFS over `<a href>` tags prevents you from missing chapters or API reference sections. Keep a visited-URL set to avoid cycles.

### Retries with Backoff

Your embedding API has rate limits even when your scraper does not. Wrap async calls in exponential backoff:

Wrap `index_chunks` calls: `await with_retry(lambda: index_chunks(chunks, index))`.

* * *

## Production Checklist

Before running this at scale, verify:

-   **Freshness TTL**: Set an expiry on indexed documents. Re-scrape on a schedule. Stale RAG context is worse than no context—your LLM will confidently cite outdated information.
-   **Minimum chunk length**: Filter out chunks with fewer than 15 words. Stubs from tables or code snippets without context are noise at query time.
-   **Metadata completeness**: Always store `scraped_at`, `source_url`, and `section_title` in vector metadata. Your LLM needs these to generate citations users can verify.
-   **Extraction failure rate**: Monitor the share of URLs returning `no_content`. Above 5% means your source sites have unusual structure and need custom extraction rules.
-   **Concurrency limits**: Do not set `concurrency` above what your scraping tier supports. Queue excess work with Redis or a task runner rather than hammering with retries.

* * *

## Takeaway

A five-stage pipeline—fetch, extract, normalize, chunk, embed—is not over-engineering. It is the minimum required to produce input that a retrieval system can actually use.

Token waste is a symptom, not the root problem. The root problem is that HTML is a rendering format, not a content format. Every step in this pipeline exists to close that gap.

The fetching layer is where most teams cut corners and regret it. Flaky HTML from failed bot-bypass attempts or unrendered JS propagates bad data through every downstream stage. Eliminating that variable with a dedicated scraping API means your engineering time goes where it compounds: extraction heuristics, chunking strategy, and retrieval evaluation.