---
source_url: "https://docs.crawl4ai.com/core/content-selection/"
title: Content Selection - Crawl4AI Documentation (v0.9.x)
mirrored_at: 2026-08-29T03:03:01.927Z
host: docs.crawl4ai.com
cited_in_42a: true
mirror_canonical: "https://index.42a.ai/docs.crawl4ai.com/core/content-selection/index"
---

> **Original source:** https://docs.crawl4ai.com/core/content-selection/

Crawl4AI provides multiple ways to **select**, **filter**, and **refine** the content from your crawls. Whether you need to target a specific CSS region, exclude entire tags, filter out external links, or remove certain domains and images, **`CrawlerRunConfig`** offers a wide range of parameters.

Below, we show how to configure these parameters and combine them for precise control.

* * *

## 1\. CSS-Based Selection

There are two ways to select content from a page: using `css_selector` or the more flexible `target_elements`.

### 1.1 Using `css_selector`

A straightforward way to **limit** your crawl results to a certain region of the page is **`css_selector`** in **`CrawlerRunConfig`**:

`import asyncio from crawl4ai import AsyncWebCrawler, CrawlerRunConfig async def main():     config = CrawlerRunConfig(         # e.g., first 30 items from Hacker News         css_selector=".athing:nth-child(-n+30)"       )     async with AsyncWebCrawler() as crawler:         result = await crawler.arun(             url="https://news.ycombinator.com/newest",              config=config         )         print("Partial HTML length:", len(result.cleaned_html)) if __name__ == "__main__":     asyncio.run(main())`

**Result**: Only elements matching that selector remain in `result.cleaned_html`.

### 1.2 Using `target_elements`

The `target_elements` parameter provides more flexibility by allowing you to target **multiple elements** for content extraction while preserving the entire page context for other features:

`import asyncio from crawl4ai import AsyncWebCrawler, CrawlerRunConfig async def main():     config = CrawlerRunConfig(         # Target article body and sidebar, but not other content         target_elements=["article.main-content", "aside.sidebar"]     )     async with AsyncWebCrawler() as crawler:         result = await crawler.arun(             url="https://example.com/blog-post",              config=config         )         print("Markdown focused on target elements")         print("Links from entire page still available:", len(result.links.get("internal", []))) if __name__ == "__main__":     asyncio.run(main())`

**Key difference**: With `target_elements`, the markdown generation and structural data extraction focus on those elements, but other page elements (like links, images, and tables) are still extracted from the entire page. This gives you fine-grained control over what appears in your markdown content while preserving full page context for link analysis and media collection.

* * *

## 2\. Content Filtering & Exclusions

### 2.1 Basic Overview

`config = CrawlerRunConfig(     # Content thresholds     word_count_threshold=10,        # Minimum words per block     # Tag exclusions     excluded_tags=['form', 'header', 'footer', 'nav'],     # Link filtering     exclude_external_links=True,         exclude_social_media_links=True,     # Block entire domains     exclude_domains=["adtrackers.com", "spammynews.org"],         exclude_social_media_domains=["facebook.com", "twitter.com"],     # Media filtering     exclude_external_images=True )`

**Explanation**:

-   **`word_count_threshold`**: Ignores text blocks under X words. Helps skip trivial blocks like short nav or disclaimers.
-   **`excluded_tags`**: Removes entire tags (`<form>`, `<header>`, `<footer>`, etc.).
-   **Link Filtering**:
-   `exclude_external_links`: Strips out external links and may remove them from `result.links`.
-   `exclude_social_media_links`: Removes links pointing to known social media domains.
-   `exclude_domains`: A custom list of domains to block if discovered in links.
-   `exclude_social_media_domains`: A curated list (override or add to it) for social media sites.
-   **Media Filtering**:
-   `exclude_external_images`: Discards images not hosted on the same domain as the main page (or its subdomains).

By default in case you set `exclude_social_media_links=True`, the following social media domains are excluded:

`[     'facebook.com',     'twitter.com',     'x.com',     'linkedin.com',     'instagram.com',     'pinterest.com',     'tiktok.com',     'snapchat.com',     'reddit.com', ]`

### 2.2 Example Usage

`import asyncio from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode async def main():     config = CrawlerRunConfig(         css_selector="main.content",          word_count_threshold=10,         excluded_tags=["nav", "footer"],         exclude_external_links=True,         exclude_social_media_links=True,         exclude_domains=["ads.com", "spammytrackers.net"],         exclude_external_images=True,         cache_mode=CacheMode.BYPASS     )     async with AsyncWebCrawler() as crawler:         result = await crawler.arun(url="https://news.ycombinator.com", config=config)         print("Cleaned HTML length:", len(result.cleaned_html)) if __name__ == "__main__":     asyncio.run(main())`

**Note**: If these parameters remove too much, reduce or disable them accordingly.

* * *

## 3\. Handling Iframes

Some sites embed content in `<iframe>` tags. If you want that inline:

`config = CrawlerRunConfig(     # Merge iframe content into the final output     process_iframes=True,     remove_overlay_elements=True,     # Remove GDPR/cookie consent popups (OneTrust, Cookiebot, etc.)     remove_consent_popups=True )`

**Usage**:

`import asyncio from crawl4ai import AsyncWebCrawler, CrawlerRunConfig async def main():     config = CrawlerRunConfig(         process_iframes=True,         remove_overlay_elements=True     )     async with AsyncWebCrawler() as crawler:         result = await crawler.arun(             url="https://example.org/iframe-demo",              config=config         )         print("Iframe-merged length:", len(result.cleaned_html)) if __name__ == "__main__":     asyncio.run(main())`

* * *

## 3.1 Flattening Shadow DOM

Sites built with **Web Components** (Stencil, Lit, Shoelace, Angular Elements, etc.) render content inside [Shadow DOM](https://developer.mozilla.org/en-US/docs/Web/API/Web_components/Using_shadow_DOM) — an encapsulated sub-tree that is invisible to normal page serialization. The browser renders it on screen, but `page.content()` never includes it.

Set `flatten_shadow_dom=True` to walk all shadow trees, resolve `<slot>` projections, and produce a single flat HTML document:

`config = CrawlerRunConfig(     # Flatten shadow DOM into the main document     flatten_shadow_dom=True,     # Give web components time to hydrate     wait_until="load",     delay_before_return_html=3.0, )`

**Full example** — crawling a product page where specs live inside shadow roots:

`import asyncio from crawl4ai import AsyncWebCrawler, CrawlerRunConfig async def main():     config = CrawlerRunConfig(         flatten_shadow_dom=True,         wait_until="load",         delay_before_return_html=3.0,     )     async with AsyncWebCrawler() as crawler:         result = await crawler.arun(             url="https://store.boschrexroth.com/en/us/p/hydraulic-cylinder-r900999011",             config=config,         )         # Without flatten_shadow_dom: ~1 KB of markdown (breadcrumbs only)         # With flatten_shadow_dom:   ~33 KB (full product specs, downloads, etc.)         print(len(result.markdown.raw_markdown)) if __name__ == "__main__":     asyncio.run(main())`

When `flatten_shadow_dom=True` is set, Crawl4AI also injects an init script that force-opens **closed** shadow roots (by patching `Element.prototype.attachShadow`), so even components that use `mode: 'closed'` become accessible.

> **Tip**: Web components need JavaScript to run before they render content (a process called _hydration_). Use `wait_until="load"` and a `delay_before_return_html` of 2–5 seconds to ensure components are fully hydrated before flattening.

For a complete runnable example, see [`shadow_dom_crawling.py`](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/shadow_dom_crawling.py).

* * *

You can combine content selection with a more advanced extraction strategy. For instance, a **CSS-based** or **LLM-based** extraction strategy can run on the filtered HTML.

`import asyncio import json from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode from crawl4ai import JsonCssExtractionStrategy async def main():     # Minimal schema for repeated items     schema = {         "name": "News Items",         "baseSelector": "tr.athing",         "fields": [             {"name": "title", "selector": "span.titleline a", "type": "text"},             {                 "name": "link",                  "selector": "span.titleline a",                  "type": "attribute",                  "attribute": "href"             }         ]     }     config = CrawlerRunConfig(         # Content filtering         excluded_tags=["form", "header"],         exclude_domains=["adsite.com"],         # CSS selection or entire page         css_selector="table.itemlist",         # No caching for demonstration         cache_mode=CacheMode.BYPASS,         # Extraction strategy         extraction_strategy=JsonCssExtractionStrategy(schema)     )     async with AsyncWebCrawler() as crawler:         result = await crawler.arun(             url="https://news.ycombinator.com/newest",              config=config         )         data = json.loads(result.extracted_content)         print("Sample extracted item:", data[:1])  # Show first item if __name__ == "__main__":     asyncio.run(main())`

`import asyncio import json from pydantic import BaseModel, Field from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, LLMConfig from crawl4ai import LLMExtractionStrategy class ArticleData(BaseModel):     headline: str     summary: str async def main():     llm_strategy = LLMExtractionStrategy(         llm_config = LLMConfig(provider="openai/gpt-4",api_token="sk-YOUR_API_KEY")         schema=ArticleData.schema(),         extraction_type="schema",         instruction="Extract 'headline' and a short 'summary' from the content."     )     config = CrawlerRunConfig(         exclude_external_links=True,         word_count_threshold=20,         extraction_strategy=llm_strategy     )     async with AsyncWebCrawler() as crawler:         result = await crawler.arun(url="https://news.ycombinator.com", config=config)         article = json.loads(result.extracted_content)         print(article) if __name__ == "__main__":     asyncio.run(main())`

Here, the crawler:

-   Filters out external links (`exclude_external_links=True`).
-   Ignores very short text blocks (`word_count_threshold=20`).
-   Passes the final HTML to your LLM strategy for an AI-driven parse.

* * *

## 5\. Comprehensive Example

Below is a short function that unifies **CSS selection**, **exclusion** logic, and a pattern-based extraction, demonstrating how you can fine-tune your final data:

`import asyncio import json from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode from crawl4ai import JsonCssExtractionStrategy async def extract_main_articles(url: str):     schema = {         "name": "ArticleBlock",         "baseSelector": "div.article-block",         "fields": [             {"name": "headline", "selector": "h2", "type": "text"},             {"name": "summary", "selector": ".summary", "type": "text"},             {                 "name": "metadata",                 "type": "nested",                 "fields": [                     {"name": "author", "selector": ".author", "type": "text"},                     {"name": "date", "selector": ".date", "type": "text"}                 ]             }         ]     }     config = CrawlerRunConfig(         # Keep only #main-content         css_selector="#main-content",         # Filtering         word_count_threshold=10,         excluded_tags=["nav", "footer"],           exclude_external_links=True,         exclude_domains=["somebadsite.com"],         exclude_external_images=True,         # Extraction         extraction_strategy=JsonCssExtractionStrategy(schema),         cache_mode=CacheMode.BYPASS     )     async with AsyncWebCrawler() as crawler:         result = await crawler.arun(url=url, config=config)         if not result.success:             print(f"Error: {result.error_message}")             return None         return json.loads(result.extracted_content) async def main():     articles = await extract_main_articles("https://news.ycombinator.com/newest")     if articles:         print("Extracted Articles:", articles[:2])  # Show first 2 if __name__ == "__main__":     asyncio.run(main())`

**Why This Works**: - **CSS** scoping with `#main-content`.  
\- Multiple **exclude\_** parameters to remove domains, external images, etc.  
\- A **JsonCssExtractionStrategy** to parse repeated article blocks.

* * *

## 6\. Scraping Modes

Crawl4AI uses `LXMLWebScrapingStrategy` (LXML-based) as the default scraping strategy for HTML content processing. This strategy offers excellent performance, especially for large HTML documents.

**Note:** For backward compatibility, `WebScrapingStrategy` is still available as an alias for `LXMLWebScrapingStrategy`.

`from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, LXMLWebScrapingStrategy async def main():     # Default configuration already uses LXMLWebScrapingStrategy     config = CrawlerRunConfig()     # Or explicitly specify it if desired     config_explicit = CrawlerRunConfig(         scraping_strategy=LXMLWebScrapingStrategy()     )     async with AsyncWebCrawler() as crawler:         result = await crawler.arun(             url="https://example.com",              config=config         )`

You can also create your own custom scraping strategy by inheriting from `ContentScrapingStrategy`. The strategy must return a `ScrapingResult` object with the following structure:

`from crawl4ai import ContentScrapingStrategy, ScrapingResult, MediaItem, Media, Link, Links class CustomScrapingStrategy(ContentScrapingStrategy):     def scrap(self, url: str, html: str, **kwargs) -> ScrapingResult:         # Implement your custom scraping logic here         return ScrapingResult(             cleaned_html="<html>...</html>",  # Cleaned HTML content             success=True,                     # Whether scraping was successful             media=Media(                 images=[                      # List of images found                     MediaItem(                         src="https://example.com/image.jpg",                         alt="Image description",                         desc="Surrounding text",                         score=1,                         type="image",                         group_id=1,                         format="jpg",                         width=800                     )                 ],                 videos=[],                    # List of videos (same structure as images)                 audios=[]                     # List of audio files (same structure as images)             ),             links=Links(                 internal=[                    # List of internal links                     Link(                         href="https://example.com/page",                         text="Link text",                         title="Link title",                         base_domain="example.com"                     )                 ],                 external=[]                   # List of external links (same structure)             ),             metadata={                        # Additional metadata                 "title": "Page Title",                 "description": "Page description"             }         )     async def ascrap(self, url: str, html: str, **kwargs) -> ScrapingResult:         # For simple cases, you can use the sync version         return await asyncio.to_thread(self.scrap, url, html, **kwargs)`

### Performance Considerations

The LXML strategy provides excellent performance, particularly when processing large HTML documents, offering up to 10-20x faster processing compared to BeautifulSoup-based approaches.

Benefits of LXML strategy: - Fast processing of large HTML documents (especially >100KB) - Efficient memory usage - Good handling of well-formed HTML - Robust table detection and extraction

### Backward Compatibility

For users upgrading from earlier versions: - `WebScrapingStrategy` is now an alias for `LXMLWebScrapingStrategy` - Existing code using `WebScrapingStrategy` will continue to work without modification - No changes are required to your existing code

* * *

## 7\. Combining CSS Selection Methods

You can combine `css_selector` and `target_elements` in powerful ways to achieve fine-grained control over your output:

`import asyncio from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode async def main():     # Target specific content but preserve page context     config = CrawlerRunConfig(         # Focus markdown on main content and sidebar         target_elements=["#main-content", ".sidebar"],         # Global filters applied to entire page         excluded_tags=["nav", "footer", "header"],         exclude_external_links=True,         # Use basic content thresholds         word_count_threshold=15,         cache_mode=CacheMode.BYPASS     )     async with AsyncWebCrawler() as crawler:         result = await crawler.arun(             url="https://example.com/article",             config=config         )         print(f"Content focuses on specific elements, but all links still analyzed")         print(f"Internal links: {len(result.links.get('internal', []))}")         print(f"External links: {len(result.links.get('external', []))}") if __name__ == "__main__":     asyncio.run(main())`

This approach gives you the best of both worlds: - Markdown generation and content extraction focus on the elements you care about - Links, images and other page data still give you the full context of the page - Content filtering still applies globally

## 8\. Conclusion

By mixing **target\_elements** or **css\_selector** scoping, **content filtering** parameters, and advanced **extraction strategies**, you can precisely **choose** which data to keep. Key parameters in **`CrawlerRunConfig`** for content selection include:

1.  **`target_elements`** – Array of CSS selectors to focus markdown generation and data extraction, while preserving full page context for links and media.
2.  **`css_selector`** – Basic scoping to an element or region for all extraction processes.
3.  **`word_count_threshold`** – Skip short blocks.
4.  **`excluded_tags`** – Remove entire HTML tags.
5.  **`exclude_external_links`**, **`exclude_social_media_links`**, **`exclude_domains`** – Filter out unwanted links or domains.
6.  **`exclude_external_images`** – Remove images from external sources.
7.  **`process_iframes`** – Merge iframe content if needed.

Combine these with structured extraction (CSS, LLM-based, or others) to build powerful crawls that yield exactly the content you want, from raw or cleaned HTML up to sophisticated JSON structures. For more detail, see [Configuration Reference](https://docs.crawl4ai.com/api/parameters/). Enjoy curating your data to the max!