---
source_url: "https://alterlab.io/blog/web-scraping-pipeline-for-llm-rag-clean-markdown"
title: "Web Scraping Pipeline for LLM & RAG: Clean Markdown | AlterLab"
mirrored_at: 2026-08-28T01:01:01.522Z
host: alterlab.io
cited_in_42a: true
mirror_canonical: "https://index.42a.ai/alterlab.io/blog/web-scraping-pipeline-for-llm-rag-clean-markdown"
---

> **Original source:** https://alterlab.io/blog/web-scraping-pipeline-for-llm-rag-clean-markdown

AlterLab handles this automatically — scrape any URL with one API call. No infrastructure required.

[Try it free](https://alterlab.io/playground)

The biggest quality problem in RAG pipelines isn't the embedding model or the vector store — it's the input data. Raw HTML fed into a chunker produces token-heavy garbage: navigation menus, cookie banners, inline styles, and `<script>` blocks that dilute every embedding you generate. Clean markdown eliminates 60–80% of that noise before a single token reaches your LLM.

This post walks through a production-ready pipeline: fetch pages with bot-bypass-aware scraping, convert to structured markdown, chunk on heading boundaries, and cache aggressively to control cost.

## Why Markdown Beats Raw HTML for LLM Inputs

A typical documentation page runs 8,000+ tokens as raw HTML and around 1,200 tokens as clean markdown. That gap matters at three stages:

-   **Chunking**: HTML chunkers split on character count, slicing mid-tag, mid-sentence, and mid-function. Markdown respects the document's own semantic structure.
-   **Retrieval precision**: Boilerplate (`<nav>`, `<footer>`, repeated header text) bleeds into embedding space and degrades cosine similarity scores on meaningful content.
-   **LLM context windows**: Smaller, cleaner chunks mean more retrieved context fits in the prompt window without exceeding token limits.

The fix is to request markdown output at the fetch layer, not post-process HTML downstream.

## Pipeline Architecture

## Step 1: Fetching Pages with Anti-Bot Bypass

Most production scraping targets — documentation sites, knowledge bases, e-commerce, news publishers — run Cloudflare or similar bot detection. A raw `requests.get()` returns a 403 or a JS challenge page, neither of which contains your content.

The [AlterLab anti-bot bypass API](https://alterlab.io/anti-bot-bypass-api) handles Cloudflare, Datadome, and CAPTCHA challenges transparently. You send a URL and receive content. No fingerprint maintenance, no proxy rotation code, no challenge-solving logic on your end.

Here's the cURL equivalent to verify the endpoint before writing any application code:

The `output_format: "markdown"` parameter is the critical lever. Instead of receiving an HTML blob, you get a pre-processed markdown document with headings, fenced code blocks, and lists intact — ready for a splitter.

## Step 2: The Full Python Pipeline

The [Python SDK](https://alterlab.io/web-scraping-api-python) ships with a batteries-included client. Install dependencies, then wire up the complete ingest flow:

## Step 3: Chunking Strategy

Character-count chunking — `RecursiveCharacterTextSplitter` with `chunk_size=1000` — is fine for prose but breaks code-heavy documentation mid-function and splits conceptually related content across chunk boundaries. The right splitter depends on content type:

For documentation ingestion, heading-based splitting is the default choice. Add a `RecursiveCharacterTextSplitter` as a fallback to cap any single chunk at 6,000 tokens — some reference pages have multi-page sections under a single heading.

## Step 4: Cost Optimization with Caching

The main cost levers are scraping requests, embedding tokens, and storage queries. Both can be dramatically reduced with two caching layers:

**Layer 1 — URL-level ETag caching**: Most documentation and knowledge-base content is stable. Store `(url → etag)` after each fetch. On subsequent runs, issue a `HEAD` request first; if the ETag or `Last-Modified` header is unchanged, skip the scrape entirely.

**Layer 2 — Chunk-level deduplication**: Before embedding, SHA-256 hash each chunk's text. Check the vector store for that ID. If it exists, skip the embed call. This is the bigger cost saver for pipelines that re-ingest on a schedule.

## Step 5: Scaling with a Task Queue

For single-user tooling, the synchronous ingest above is sufficient. For pipelines ingesting thousands of URLs on a schedule, parallelize with Celery and Redis:

Keep worker concurrency aligned to your scraping API plan. AlterLab's [pricing plans](https://alterlab.io/pricing) scale with concurrent connections — over-parallelizing wastes retries; under-parallelizing wastes wall time.

## Handling Edge Cases

**JavaScript-heavy SPAs**: Set `js_render: true` and use `wait_for_selector` targeting the content container (`main`, `article`, `[role="main"]`). Waiting on `body` fires before React or Vue hydrates the actual content.

**Pagination**: After fetching, parse `rel="next"` link tags from the response metadata and enqueue subsequent pages in the same Celery task group. Store `(canonical_url, page_number)` as the vector ID to avoid collisions.

**PDF and binary content**: Check the response's `content_type` field before processing. If it's not `text/html`, route to a dedicated PDF extraction path (pdfplumber, pymupdf) rather than the markdown pipeline.

**Oversized sections**: A single API reference page converted to markdown can still produce chunks exceeding the 8,192-token embedding limit. Add a token-count guard after heading splitting and re-chunk any oversized section with `RecursiveCharacterTextSplitter(chunk_size=6000)` as a fallback.

**Dynamic content that isn't indexed**: Some content (login-gated pages, single-page apps loading data via authenticated XHR) won't yield useful markdown regardless of JS rendering. Identify these early in the pipeline and route them to session-based scraping or data-export APIs where they exist.

## Choosing an Embedding Model

The embedding model choice has a larger cost impact than most engineers expect:

Model

Dimensions

Cost / 1M tokens

MTEB Score

`text-embedding-ada-002`

1,536

$0.10

61.0

`text-embedding-3-small`

1,536

$0.02

62.3

`text-embedding-3-large`

3,072

$0.13

64.6

`nomic-embed-text` (local)

768

$0.00

62.4

`BGE-M3` (local)

1,024

$0.00

63.8

For most RAG use cases, `text-embedding-3-small` matches or beats `ada-002` at 5× lower cost. For air-gapped or high-volume deployments, a locally-hosted model eliminates per-token costs entirely at the price of infrastructure overhead.

## Takeaways

-   **Request markdown at the fetch layer.** Post-processing HTML is expensive and lossy; let the scraping API handle conversion before the content hits your pipeline.
-   **Split on structure, not character count.** `MarkdownHeaderTextSplitter` preserves the semantic unit of documentation. Add a token-limit fallback for oversized sections.
-   **Cache at two levels.** ETag-based URL caching prevents unnecessary re-scraping; chunk-level SHA-256 deduplication prevents unnecessary re-embedding. Together they cut ongoing costs by 90%+ on stable knowledge bases.
-   **Match worker concurrency to your API tier.** Excess parallelism hits rate limits and burns retries; it does not improve throughput.
-   **Pick the right embedding model.** `text-embedding-3-small` is the default right choice for hosted inference. Local BGE-M3 or nomic-embed-text is the right choice above ~500M tokens/month.