---
source_url: "https://apify.com/datapilot/rag-web-browser-scraper"
title: RAG Web Browser Scraper · Apify
mirrored_at: 2026-09-03T01:03:26.633Z
host: apify.com
cited_in_42a: true
mirror_canonical: "https://index.42a.ai/apify.com/datapilot/rag-web-browser-scraper"
---

> **Original source:** https://apify.com/datapilot/rag-web-browser-scraper

🚀 **RAG Web Browser Scraper** is a comprehensive Apify Actor designed to scrape web content for **RAG (Retrieval-Augmented Generation) applications**. This tool provides complete **RAG Web functionality**, including search engine results, web page crawling, and markdown conversion for any URL or search query. Whether you're building **RAG applications**, conducting web research, or extracting content for AI models, the RAG Web delivers clean, structured **RAG Web data** efficiently.

With browser automation using , the RAG Web ensures accurate extraction of dynamic web content that may not be available through simple HTTP requests. It focuses on key **RAG Web metrics** like markdown content, metadata, and HTTP status, making it an essential tool for **RAG Web analysis** and content extraction.

## 🔥 Features

-   **Comprehensive RAG Web Extraction** – Scrapes detailed **RAG Web data** from both search queries and direct URLs, including titles, descriptions, and full markdown content.
-   **Search Engine Integration** – Uses Bing to search for **RAG Web results** based on query keywords.
-   **Markdown Conversion** – Converts HTML content to markdown format for easy use in **RAG applications** and AI models.
-   **Browser Automation** – Uses for headless browser navigation to access fully rendered **RAG Web content**.
-   **Batch Processing** – Processes multiple inputs (URLs or queries) in a single run for efficient **RAG Web data collection**.
-   **Metadata Enrichment** – Provides **RAG Web metadata** like title, description, language code, and HTTP status.
-   **Residential Proxy Support** – Utilizes Apify's residential proxies to bypass restrictions and ensure high success rates for **RAG Web**.
-   **Error Handling** – Robust logging and fallback mechanisms for failed **RAG Web scrapes**.
-   **Dataset Integration** – Automatically uploads **RAG Web data** to your Apify dataset for easy export and analysis.

* * *

## ⚙️ How It Works

RAG Web Browser Scraper The RAG Web takes one or more inputs (either URLs or search queries) as input and uses to launch a headless browser. For search queries, it performs Bing searches and retrieves top results; for direct URLs, it crawls the pages directly. The scraper converts HTML to markdown, extracts metadata, and returns structured **data**. It returns detailed **RAG Web information** on success or error details on failure, providing a reliable way to gather web content for **RAG applications**.

**Key Processing Steps:**

1.  **Input Parsing** – Parse and validate URLs or search queries
2.  **Search/Navigation** – Perform Bing searches or navigate to URLs
3.  **Content Extraction** – Extract HTML and metadata from pages
4.  **Markdown Conversion** – Convert HTML to clean markdown format
5.  **Metadata Enrichment** – Extract title, description, language, status
6.  **Data Compilation** – Aggregate all extracted data
7.  **Export** – Push results to dataset in JSON format

**Key benefits for RAG Web analysis:**

-   Extract clean markdown content for **RAG and AI applications**.
-   Access web content including dynamic JavaScript-rendered pages.
-   Build **RAG Web databases** for content extraction and research.
-   Prepare structured data for Large Language Models (LLMs).
-   Create comprehensive web content datasets.

* * *

## 📥 Input

The scraper accepts the following input parameters:

Field

Type

Default

Description

`queries`

string

required

List of search queries or URLs to scrape RAG Web from, comma-separated (e.g., `"https://example.com, keyword search"`).

`maxResults`

integer

`5`

Maximum number of search results per query to process (1-20).

`useApifyProxy`

boolean

`true`

Enable residential proxies for RAG Web scraping.

`apifyProxyGroups`

array

`["RESIDENTIAL"]`

Proxy groups to use (e.g., `["RESIDENTIAL"]`).

**Example input JSON:**

```
{  "queries": "https://example.com, artificial intelligence news",  "maxResults": 10,  "useApifyProxy": true,  "apifyProxyGroups": ["RESIDENTIAL"]}
```

* * *

## 📤 Output

The scraper outputs detailed **RAG Web data** in JSON format for each input. Each record includes:

Field

Type

Description

`crawl`

object

RAG Web crawl information.

`searchResult`

object

RAG Web search result details (for queries).

`metadata`

object

RAG Web page metadata.

`markdown`

string

Converted markdown content from the page.

**Crawl Object Fields:**

Field

Type

Description

`httpStatusCode`

integer

HTTP status code of the request.

`httpStatusMessage`

string

HTTP status message.

`loadedAt`

string

ISO timestamp of page load.

`uniqueKey`

string

Unique identifier for the result.

`requestStatus`

string

Status of request (`"handled"` or `"failed"`).

**SearchResult Object Fields:**

Field

Type

Description

`title`

string

Title of the search result.

`description`

string

Description of the search result.

`url`

string

URL of the search result.

**Metadata Object Fields:**

Field

Type

Description

`title`

string

Page title.

`description`

string

Page description/meta description.

`languageCode`

string

Language code of the page (e.g., `"en"`).

`url`

string

Original URL.

**Example output for RAG Web Scraper:**

```
{  "crawl": {    "httpStatusCode": 200,    "httpStatusMessage": "OK",    "loadedAt": "2025-02-14T12:00:00.123Z",    "uniqueKey": "abc123xyz",    "requestStatus": "handled"  },  "searchResult": {    "title": "Example Page Title",    "description": "This is an example description",    "url": "https://www.example.com"  },  "metadata": {    "title": "Example Page Title",    "description": "This is an example description",    "languageCode": "en",    "url": "https://www.example.com"  },  "markdown": "# Example Title\n\nThis is the main content of the page converted to markdown format..."}
```

**Example error response:**

```
{  "url": "https://www.example.com/invalid",  "requestStatus": "failed",  "error": "Page not accessible or failed to load",  "httpStatusCode": 404}
```

* * *

## 🧰 Technical Stack

-   **Search Integration:** Bing Search API for query processing
-   **Markdown Conversion:** HTML to Markdown converter
-   **Metadata Extraction:** BeautifulSoup – HTML parsing
-   **Proxy Support:** Apify Proxy with RESIDENTIAL support
-   **Platform:** Apify Actor – serverless, scalable, integrated with Dataset and Key‑Value Store
-   **Deployment:** One‑click run on Apify Console or via REST API

* * *

## 🎯 Use Cases

-   **RAG Application Development** – Extract and prepare web content for Retrieval-Augmented Generation (RAG) systems.
-   **LLM Training Data** – Gather clean markdown content for training large language models.
-   **Web Content Extraction** – Extract structured content from websites for AI applications.
-   **Knowledge Base Building** – Build knowledge bases for chatbots and AI assistants.
-   **Research Content Aggregation** – Aggregate research content from multiple web sources.
-   **Documentation Compilation** – Compile documentation from web sources into markdown.
-   **Content Repository** – Create repositories of web content for AI systems.
-   **Information Retrieval** – Build systems for efficient information retrieval.
-   **Web Data Mining** – Mine web data for analysis and research.
-   **Dynamic Content Handling** – Extract JavaScript-rendered content accurately.
-   **Multi-Source Integration** – Combine data from multiple sources into unified formats.
-   **SEO Content Analysis** – Analyze web content structure and metadata.
-   **Web Research** – Conduct comprehensive web research with structured data extraction.
-   **Data Preparation** – Prepare data for machine learning and AI models.

* * *

## 🚀 Quick Start

1.  **Open in Apify Console** – visit the Actor page and click Try for free.
2.  **Enter queries or URLs** – provide URLs or search queries separated by commas.
3.  **Set result limits** – choose maximum results per query (1-20).
4.  **Enable proxies** – enabled by default for reliable scraping.
5.  **Click Start** – the Actor will scrape and convert content to markdown.
6.  **View Results** – check the dataset for extracted content and metadata.
7.  **Copy Markdown** – use extracted markdown content for RAG/AI applications.
8.  **Export** – download the results as JSON, CSV, or Excel.

You can also call this Actor programmatically via Apify SDK or REST API – ideal for automated content extraction and RAG pipeline integration.

* * *

## 💎 Why This Scraper?

Feature

Benefit

✅ Markdown format

Output ready for RAG and LLM applications.

✅ Search + URLs

Handle both search queries and direct URLs.

✅ Dynamic content

Extract JavaScript-rendered content.

✅ Metadata included

Get title, description, language info.

✅ Batch processing

Process multiple inputs efficiently.

✅ HTTP status info

Know page accessibility and status.

✅ Residential proxies

Bypass restrictions – reliable access.

✅ Apify ecosystem

Seamless integration with other Actors, triggers, and webhooks.

## 📦 Changelog

-   Initial release of RAG Web Scraper
-   Comprehensive web content extraction from URLs
-   Search query integration with Bing
-   HTML to markdown conversion
-   Metadata extraction (title, description, language)
-   Dynamic content handling
-   Batch processing for multiple inputs
-   HTTP status code tracking
-   Unique key generation for results
-   Residential proxy support
-   Error handling with fallback mechanisms
-   Automatic dataset integration
-   Full Apify Actor integration

* * *

## 🧑‍💻 Support & Feedback

-   **Issues & Ideas:** Open a ticket on the Apify Actor issue tracker
-   **Documentation:** Visit Apify Docs for comprehensive platform guides
-   **Community:** Join the Apify community forum for discussions and support
-   **Bug Reports:** Submit detailed bug reports through the issue tracker
-   **Feature Requests:** Suggest new features to improve the scraper

* * *

## 💰 Pricing

-   **Free** for basic usage on Apify platform
-   **Paid plans** available for higher limits and priority support
-   **Proxy credits** consumed based on residential proxy usage
-   **Compute credits** consumed for browser automation and processing

**Disclaimer:** RAG Web is provided as-is for research and AI application development purposes. Users are responsible for ensuring their usage complies with website policies and applicable laws. Markdown conversion accuracy depends on HTML structure complexity.

* * *

## 🎉 Get Started Today

**Begin extracting web content for RAG applications now!**

Use RAG Web for:

-   🤖 RAG Development
-   📚 Knowledge Base Building
-   🔍 Web Content Extraction
-   📊 Data Preparation
-   💡 AI Training

**Perfect for:**

-   AI Engineers
-   Researchers
-   Data Scientists
-   Content Strategists
-   Developers

* * *

**Last Updated:** February 2025  
**Version:** 1.0.0  
**Status:** Active Development  
**Support:** 24/7 Customer Support Available  
**Platform:** Apify

* * *

For comprehensive web research and content extraction, explore our full suite of tools:

-   Download HTML from URLs
-   Google Search Results Scraper
-   Ranked Keywords Scraper with SEO Metrics
-   All-in-One Media Downloader
-   Ultimate Video Info Fetcher