---
source_url: "https://github.com/olostep-api"
title: Olostep · GitHub
mirrored_at: 2026-08-24T15:03:25.529Z
host: github.com
cited_in_42a: true
mirror_canonical: "https://index.42a.ai/github.com/olostep-api"
---

> **Original source:** https://github.com/olostep-api

## Olostep

### Extract clean, structured data from any website — in real time.

**The web scraping, crawling, and search API powering the world's leading AI agents and companies.**

   

* * *

## **Olostep**

**Extract clean, structured data from any website — in real time.**

[**Olostep**](https://www.olostep.com/) is an API that scrapes, crawls, searches, and answers questions from the web — powering AI agents and apps with real-time, structured data.

* * *

## Why Olostep?

-   **LLM-ready output**: Clean markdown, structured JSON, screenshots, HTML, and more
-   **Real-time extraction**: No caching, no stale data — live results every time
-   **Handles the hard stuff**: JavaScript rendering, anti-bot bypasses, and dynamic content
-   **Built for AI agents**: Schema-aligned extraction and natural language answers grounded on live web data
-   **Cost-effective at scale**: The most reliable and affordable web data API on the market

* * *

## Quick Start

Sign up at [olostep.com](https://www.olostep.com/) to get your API key and start extracting data in seconds. Try the [playground](https://www.olostep.com/playground) to test it out.

### Make Your First API Request

curl -X POST 'https://api.olostep.com/v1/scrapes' \\
  -H 'Authorization: Bearer $OLOSTEP\_API\_KEY' \\
  -H 'Content-Type: application/json' \\
  -d '{"url\_to\_scrape": "https://example.com", "formats": \["markdown"\]}'

Response:

{
  "id": "scrape\_bgfa9f9wim",
  "object": "scrape",
  "created": 1774350584,
  "url": "https://example.com",
  "retrieve\_id": "bgfa9f9wim",
  "result": {
    "markdown\_content": "Example Domain\\n\\nExample Domain\\n\==============\\n\\nThis domain is for use in documentation examples without needing permission. Avoid use in operations.\\n\\n\[Learn more\](https://iana.org/domains/example)",
    "html\_content": null,
    "text\_content": null,
    "json\_content": null,
    "screenshot\_hosted\_url": null,
    "links\_on\_page": \[\],
    "page\_metadata": {
      "status\_code": 200
    }
  },
  "credits\_consumed": 1
}

* * *

## Feature Overview

Feature

Description

Docs

[**Scrapes**](#scrapes)

Convert any URL to markdown, HTML, screenshots, or structured JSON

[→](https://docs.olostep.com/features/scrapes/scrapes)

[**Crawls**](#crawls)

Scrape all pages of a website with a single request

[→](https://docs.olostep.com/features/crawls/crawls)

[**Maps**](#maps)

Discover all URLs on a website instantly

[→](https://docs.olostep.com/features/maps/maps)

[**Batches**](#batches)

Process up to 10,000 URLs in one async job

[→](https://docs.olostep.com/features/batches/batches)

[**Answers**](#answers)

Ask a question in natural language; get a source-backed, structured answer from live web data

[→](https://docs.olostep.com/features/answers/answers)

[**Searches**](#searches)

Search the web with a plain-English query — deduplicated links with titles and descriptions

[→](https://docs.olostep.com/searches/searches)

[**Agents**](#agents)

Autonomous research agents that automate data pipelines and deliver structured results on a schedule

[→](https://docs.olostep.com/features/agents/agents)

* * *

## Scrapes

Convert any URL to clean markdown, HTML, plain text, structured JSON, or a screenshot. Handles JavaScript-rendered pages, anti-bot challenges, and dynamic content automatically.

curl -X POST 'https://api.olostep.com/v1/scrapes' \\
  -H 'Authorization: Bearer $OLOSTEP\_API\_KEY' \\
  -H 'Content-Type: application/json' \\
  -d '{
    "url\_to\_scrape": "https://docs.olostep.com/get-started/welcome",
    "formats": \["markdown", "html"\]
  }'

Response:

{
  "id": "scrape\_309i70dx3x",
  "object": "scrape",
  "created": 1774366419,
  "url": "https://docs.olostep.com/get-started/welcome",
  "retrieve\_id": "309i70dx3x",
  "result": {
    "markdown\_content": "Welcome to Olostep - Olostep Docs\\n\\nOlostep is a web scraping, crawling, and search API...",
    "html\_content": "<html lang=\\"en\\" class=\\"\_\_variable\_47c970 ...\\"\>...",
    "text\_content": null,
    "json\_content": null,
    "screenshot\_hosted\_url": null,
    "html\_hosted\_url": "https://olostep-storage.s3.us-east-1.amazonaws.com/text\_309i70dx3x.txt",
    "markdown\_hosted\_url": "https://olostep-storage.s3.us-east-1.amazonaws.com/markDown\_309i70dx3x.txt",
    "links\_on\_page": \[\],
    "page\_metadata": {
      "status\_code": 200
    }
  },
  "credits\_consumed": 1,
  "metadata": {}
}

### Extract Structured Data

You can extract structured JSON in two ways: **using Parsers** or **LLM extraction**.

#### Using a Parser (recommended for scale)

Parsers turn unstructured web data into structured JSON. They are ideal when you need data at scale in a recurrent way from the same websites — significantly more cost-efficient (1–5 credits) than LLM extraction (20 credits). Olostep offers [pre-built parsers](https://www.olostep.com/store) for popular websites, and you can also create your own through the [dashboard](https://www.olostep.com/dashboard/parsers).

Define `formats: ["json"]` and provide a `parser` `id`:

curl -X POST 'https://api.olostep.com/v1/scrapes' \\
  -H 'Authorization: Bearer $OLOSTEP\_API\_KEY' \\
  -H 'Content-Type: application/json' \\
  -d '{
    "url\_to\_scrape": "https://www.google.com/search?q=olostep+web+scraping+api&gl=us&hl=en",
    "formats": \["json"\],
    "parser": {
      "id": "@olostep/google-search"
    }
  }'

Response:

{
  "id": "scrape\_yxhfnrlfuh",
  "object": "scrape",
  "created": 1774369250,
  "url": "https://www.google.com/search?q=olostep+web+scraping+api&gl=us&hl=en",
  "retrieve\_id": "yxhfnrlfuh",
  "result": {
    "html\_content": null,
    "markdown\_content": null,
    "text\_content": null,
    "json\_content": "{\\"searchParameters\\":{\\"q\\":\\"olostep web scraping api\\",\\"gl\\":\\"us\\",\\"hl\\":\\"en\\",\\"type\\":\\"search\\",\\"engine\\":\\"google\\"},\\"organic\\":\[{\\"title\\":\\"Olostep - Web Data API for AI, Crawling & Data Extraction\\",\\"link\\":\\"https://www.olostep.com/\\",\\"snippet\\":\\"Olostep is a Web Data API that helps AI teams search, crawl, scrape and structure web data through a single, developer-friendly platform. Built for modern AI ...\\",\\"position\\":1},{\\"title\\":\\"Olostep Docs: Welcome to Olostep\\",\\"link\\":\\"https://docs.olostep.com/get-started/welcome\\",\\"snippet\\":\\"The Olostep API is the best web search, scraping and crawling API for AI used by some of the leading startups in the world.\\",\\"position\\":2},{\\"title\\":\\"Scrape - Olostep Docs\\",\\"link\\":\\"https://docs.olostep.com/features/scrapes/scrapes\\",\\"snippet\\":\\"Through the Olostep /v1/scrapes endpoint you can extract LLM-friendly Markdown, HTML, text, screenshots, or structured JSON from any URL in real time.\\",\\"position\\":3}\],\\"relatedSearches\\":\[{\\"query\\":\\"Olostep web scraping api tutorial\\"},{\\"query\\":\\"Olostep pricing\\"}\]}",
    "screenshot\_hosted\_url": null,
    "json\_hosted\_url": "https://olostep-storage.s3.us-east-1.amazonaws.com/json\_yxhfnrlfuh.json",
    "links\_on\_page": \[\],
    "page\_metadata": {
      "status\_code": 200
    }
  },
  "credits\_consumed": 2,
  "metadata": {}
}

Available pre-built parsers include `@olostep/google-search`, `@olostep/amazon-it-product`, `@olostep/extract-emails`, `@olostep/extract-calendars`, and `@olostep/extract-socials`. Parsers are self-healing and update automatically when websites change. Need a custom parser? Contact [info@olostep.com](mailto:info@olostep.com).

#### Using LLM Extraction (schema and/or prompt)

For websites with changing structures or one-off extraction needs, provide `llm_extract` with a JSON Schema (`schema`) and/or a natural language instruction (`prompt`). You can pass both parameters, but if both are provided, `schema` takes precedence.

**With a schema:**

curl -X POST 'https://api.olostep.com/v1/scrapes' \\
  -H 'Authorization: Bearer $OLOSTEP\_API\_KEY' \\
  -H 'Content-Type: application/json' \\
  -d '{
    "url\_to\_scrape": "https://en.wikipedia.org/wiki/Web\_scraping",
    "formats": \["markdown", "json"\],
    "llm\_extract": {
      "schema": {
        "type": "object",
        "properties": {
          "title": {"type": "string"},
          "summary": {"type": "string"}
        }
      }
    }
  }'

Response:

{
  "id": "scrape\_afuw1y7lm0",
  "object": "scrape",
  "created": 1774369369,
  "url": "https://en.wikipedia.org/wiki/Web\_scraping",
  "retrieve\_id": "afuw1y7lm0",
  "result": {
    "markdown\_content": "Web scraping - Wikipedia\\n\\nWeb scraping\\n\============\\n\\n...",
    "json\_content": "{\\"summary\\":\\"Web scraping, web harvesting, or web data extraction is data scraping used for extracting data from websites. Web scraping software may directly access the World Wide Web using the Hypertext Transfer Protocol or a web browser. While web scraping can be done manually by a software user, the term typically refers to automated processes implemented using a bot or web crawler.\\",\\"title\\":\\"Web scraping\\"}",
    "html\_content": null,
    "text\_content": null,
    "screenshot\_hosted\_url": null,
    "markdown\_hosted\_url": "https://olostep-storage.s3.us-east-1.amazonaws.com/markDown\_afuw1y7lm0.txt",
    "json\_hosted\_url": "https://olostep-storage.s3.us-east-1.amazonaws.com/json\_afuw1y7lm0.json",
    "links\_on\_page": \[\],
    "page\_metadata": {
      "status\_code": 200
    }
  },
  "credits\_consumed": 20,
  "metadata": {}
}

> **Note:** `result.json_content` returns a stringified JSON. Parse it in your code to access the structured data.

**With a prompt (no schema):**

If you just pass a `prompt`, the LLM will extract the data based on the prompt and decide the data structure on its own:

curl -X POST 'https://api.olostep.com/v1/scrapes' \\
  -H 'Authorization: Bearer $OLOSTEP\_API\_KEY' \\
  -H 'Content-Type: application/json' \\
  -d '{
    "url\_to\_scrape": "https://docs.olostep.com/get-started/welcome",
    "formats": \["json"\],
    "llm\_extract": {
      "prompt": "Extract the product name and a list of all API endpoint names mentioned on this page."
    }
  }'

Response:

{
  "id": "scrape\_ub9d81eqsw",
  "object": "scrape",
  "created": 1774369391,
  "url": "https://docs.olostep.com/get-started/welcome",
  "retrieve\_id": "ub9d81eqsw",
  "result": {
    "html\_content": null,
    "markdown\_content": null,
    "text\_content": null,
    "json\_content": "{\\"productName\\":\\"Olostep\\",\\"endpoints\\":\[\\"/scrapes\\",\\"/crawls\\",\\"/maps\\",\\"/batches\\",\\"/answers\\",\\"/parsers\\",\\"/agents\\",\\"/files\\",\\"/sandboxes\\"\]}",
    "screenshot\_hosted\_url": null,
    "json\_hosted\_url": "https://olostep-storage.s3.us-east-1.amazonaws.com/json\_ub9d81eqsw.json",
    "links\_on\_page": \[\],
    "page\_metadata": {
      "status\_code": 200
    }
  },
  "credits\_consumed": 20,
  "metadata": {}
}

### Screenshot

curl -X POST 'https://api.olostep.com/v1/scrapes' \\
  -H 'Authorization: Bearer $OLOSTEP\_API\_KEY' \\
  -H 'Content-Type: application/json' \\
  -d '{
    "url\_to\_scrape": "https://www.olostep.com",
    "formats": \["screenshot"\]
  }'

Response:

{
  "id": "scrape\_bj3tz6vang",
  "object": "scrape",
  "created": 1774366431,
  "url": "https://www.olostep.com",
  "retrieve\_id": "bj3tz6vang",
  "result": {
    "screenshot\_hosted\_url": "https://olostep-screenshots.s3.us-east-1.amazonaws.com/image\_bj3tz6vang.png",
    "markdown\_content": null,
    "html\_content": null,
    "page\_metadata": {
      "status\_code": 200
    }
  },
  "credits\_consumed": 1,
  "metadata": {}
}

### Actions (Interact Before Scraping)

Click, type, scroll, and wait before extracting — for login-gated pages and JS-heavy SPAs:

import os
from olostep import Olostep

client \= Olostep(api\_key\=os.environ\["OLOSTEP\_API\_KEY"\])

result \= client.scrapes.create(
    url\_to\_scrape\="https://example.com/dashboard",
    formats\=\["markdown"\],
    actions\=\[
        {"type": "click",      "selector": "#login-button"},
        {"type": "fill\_input", "selector": "#email",    "value": "user@example.com"},
        {"type": "fill\_input", "selector": "#password", "value": "password"},
        {"type": "click",      "selector": "button\[type='submit'\]"},
        {"type": "wait",       "milliseconds": 2000}
    \]
)
print(result.result.markdown\_content)

### Scrape Formats

Available formats: `markdown`, `html`, `text`, `json`, `screenshot`, `raw_pdf`

Format

Response Field

Description

`markdown`

`markdown_content`

Clean markdown — ideal for LLM context

`html`

`html_content`

Full rendered HTML

`text`

`text_content`

Plain text, no markup

`json`

`json_content`

Structured JSON (requires `llm_extract` or `parser`)

`screenshot`

`screenshot_hosted_url`

Full-page screenshot URL (PNG)

`raw_pdf`

`file_hosted_url`

Raw PDF text extraction

* * *

## Crawls

Crawl an entire website and get content from all pages.

curl -X POST 'https://api.olostep.com/v1/crawls' \\
  -H 'Authorization: Bearer $OLOSTEP\_API\_KEY' \\
  -H 'Content-Type: application/json' \\
  -d '{
    "start\_url": "https://docs.olostep.com",
    "max\_pages": 5,
    "include\_urls": \["/features/\*\*", "/get-started/\*\*"\],
    "exclude\_urls": \["/changelog/\*\*"\],
    "max\_depth": 2
  }'

Returns a job ID:

{
  "id": "crawl\_pznr94twat",
  "object": "crawl",
  "status": "in\_progress",
  "created": 1774366526357,
  "start\_date": "2026-03-24",
  "start\_url": "https://docs.olostep.com",
  "max\_pages": 5,
  "max\_depth": 2,
  "exclude\_urls": \["/changelog/\*\*"\],
  "include\_urls": \["/features/\*\*", "/get-started/\*\*"\],
  "include\_external": false,
  "current\_depth": 0,
  "pages\_count": 0
}

### Check Crawl Pages

curl -X GET 'https://api.olostep.com/v1/crawls/crawl\_pznr94twat/pages' \\
  -H 'Authorization: Bearer $OLOSTEP\_API\_KEY'

{
  "id": "crawl\_pznr94twat",
  "object": "crawl",
  "status": "completed",
  "pages\_count": 5,
  "pages": \[
    {
      "id": "urlId\_1782f5bad2",
      "retrieve\_id": "x9pkauadjv",
      "url": "https://docs.olostep.com/",
      "is\_external": false
    },
    {
      "id": "urlId\_c0233df380",
      "retrieve\_id": "xpqm6uul08",
      "url": "https://docs.olostep.com/get-started/authentication",
      "is\_external": false
    },
    {
      "id": "urlId\_a2f0d23899",
      "retrieve\_id": "lltjltz9hn",
      "url": "https://docs.olostep.com/features/answers/answers",
      "is\_external": false
    },
    {
      "id": "urlId\_36035aafd9",
      "retrieve\_id": "hkaxd1zlng",
      "url": "https://docs.olostep.com/features/maps/maps",
      "is\_external": false
    },
    {
      "id": "urlId\_c0a2f0feb1",
      "retrieve\_id": "owliom9ldx",
      "url": "https://docs.olostep.com/features/crawls/crawls",
      "is\_external": false
    }
  \],
  "metadata": {
    "external\_urls": \[\],
    "failed\_urls": \[\]
  }
}

### Retrieve Page Content

curl -X GET 'https://api.olostep.com/v1/retrieve?retrieve\_id=x9pkauadjv&formats=markdown' \\
  -H 'Authorization: Bearer $OLOSTEP\_API\_KEY'

{
  "markdown\_content": "Welcome to Olostep - Olostep Docs\\n\\nOlostep is a web scraping, crawling, and search API...",
  "html\_content": null,
  "json\_content": null,
  "screenshot\_hosted\_url": null,
  "markdown\_hosted\_url": "https://olostep-storage.s3.amazonaws.com/markDown\_x9pkauadjv.txt",
  "success": true
}

**Note:** The SDKs handle polling automatically for a better developer experience.

* * *

## Maps

Discover all URLs on a website instantly.

curl -X POST 'https://api.olostep.com/v1/maps' \\
  -H 'Authorization: Bearer $OLOSTEP\_API\_KEY' \\
  -H 'Content-Type: application/json' \\
  -d '{"url": "https://docs.olostep.com", "top\_n": 10}'

Response:

{
  "id": "map\_6ahdf6sf5o",
  "urls\_count": 10,
  "urls": \[
    "https://docs.olostep.com/sdks/python",
    "https://docs.olostep.com/examples/batch",
    "https://docs.olostep.com/integrations/relay",
    "https://docs.olostep.com/integrations/mastra",
    "https://docs.olostep.com/integrations/n8n",
    "https://docs.olostep.com/concepts/latency",
    "https://docs.olostep.com/get-started/welcome",
    "https://docs.olostep.com/integrations/mcp-server",
    "https://docs.olostep.com/features/maps/maps",
    "https://docs.olostep.com/de/integrations/mastra"
  \]
}

### Map with URL Filters

Find specific URLs within a site using glob patterns:

import os
from olostep import Olostep

client \= Olostep(api\_key\=os.environ\["OLOSTEP\_API\_KEY"\])

result \= client.maps.create(
    url\="https://docs.olostep.com",
    include\_urls\=\["/features/\*\*"\],
    exclude\_urls\=\["/changelog/\*\*"\],
    top\_n\=50
)

for url in result.urls:
    print(url)

\# Paginate for large sites
if result.cursor:
    next\_page \= client.maps.create(
        url\="https://docs.olostep.com",
        cursor\=result.cursor
    )

* * *

## Batches

Scrape multiple URLs at once — up to 10,000 URLs in a single job, completing in 5–8 minutes.

curl -X POST 'https://api.olostep.com/v1/batches' \\
  -H 'Authorization: Bearer $OLOSTEP\_API\_KEY' \\
  -H 'Content-Type: application/json' \\
  -d '{
    "items": \[
      {"custom\_id": "item-001", "url": "https://docs.olostep.com/features/scrapes/scrapes"},
      {"custom\_id": "item-002", "url": "https://docs.olostep.com/features/crawls/crawls"},
      {"custom\_id": "item-003", "url": "https://docs.olostep.com/features/answers/answers"}
    \],
    "country": "us"
  }'

Returns a job ID:

{
  "id": "batch\_uc2q18nijo",
  "object": "batch",
  "status": "in\_progress",
  "created": 1774366558223,
  "total\_urls": 3,
  "completed\_urls": 0,
  "batch\_country": "us",
  "country": "us",
  "start\_date": "2026-03-24",
  "metadata": {}
}

### Check Batch Status

curl -X GET 'https://api.olostep.com/v1/batches/batch\_uc2q18nijo/items' \\
  -H 'Authorization: Bearer $OLOSTEP\_API\_KEY'

{
  "id": "batch\_uc2q18nijo",
  "object": "batch",
  "status": "completed",
  "items": \[
    {
      "custom\_id": "item-002",
      "retrieve\_id": "uc2q18nijo\_item-002",
      "url": "https://docs.olostep.com/features/crawls/crawls"
    },
    {
      "custom\_id": "item-003",
      "retrieve\_id": "uc2q18nijo\_item-003",
      "url": "https://docs.olostep.com/features/answers/answers"
    },
    {
      "custom\_id": "item-001",
      "retrieve\_id": "uc2q18nijo\_item-001",
      "url": "https://docs.olostep.com/features/scrapes/scrapes"
    }
  \],
  "items\_count": 3
}

### Retrieve Item Content

curl -X GET 'https://api.olostep.com/v1/retrieve?retrieve\_id=uc2q18nijo\_item-002&formats=markdown' \\
  -H 'Authorization: Bearer $OLOSTEP\_API\_KEY'

{
  "markdown\_content": "Crawl - Olostep Docs\\n\\nCrawl\\n\\nCrawl an entire website and get content from all pages...",
  "html\_content": null,
  "json\_content": null,
  "screenshot\_hosted\_url": null,
  "success": true,
  "markdown\_hosted\_url": "https://olostep-storage.s3.amazonaws.com/markDown\_uc2q18nijo\_item-002.txt"
}

**Note:** The SDKs handle polling automatically for a better developer experience.

* * *

## Answers

> **Olostep's unique differentiator.** Ask a question in natural language and get a source-backed answer synthesized from live web data. No other web data API exposes this — ideal for grounding LLM agents on real-time information without managing search and scrape pipelines yourself.

curl -X POST 'https://api.olostep.com/v1/answers' \\
  -H 'Authorization: Bearer $OLOSTEP\_API\_KEY' \\
  -H 'Content-Type: application/json' \\
  -d '{"task": "What is Olostep and what are its main features?"}'

Response:

{
  "id": "answer\_jg4inrka6b",
  "object": "answer",
  "created": 1774366472,
  "metadata": {},
  "task": "What is Olostep and what are its main features?",
  "result": {
    "json\_content": "{\\"result\\":\\"Olostep is a Web Data API that lets AI teams search, crawl, scrape and structure real-time web data through a single developer-friendly platform.\\\\n\\\\nMain features:\\\\n- Scrape any URL for Markdown, HTML, text or structured JSON\\\\n- Crawl all subpages of a site without needing a sitemap\\\\n- Batch process up to 10,000 URLs concurrently\\\\n- Ask natural-language questions and get AI answers with sources via /answers\\\\n- Create custom parsers to turn unstructured data into clean JSON\\\\n- Build, schedule and run research agents with no-code prompts\\\\n- Full JavaScript rendering and premium residential proxies on every request\\\\n- Multi-format outputs (JSON, Markdown, HTML, PDF) and automated form-fill actions\\"}",
    "json\_hosted\_url": "https://olostep-storage.s3.us-east-1.amazonaws.com/answer\_jg4inrka6b.json",
    "sources": \[
      "https://github.com/olostep",
      "https://docs.olostep.com/get-started/welcome",
      "https://www.olostep.com/"
    \]
  }
}

### Structured JSON Extraction with Schema

Provide a `json` parameter with empty values as a schema to guide the output. The API will return structured data matching your schema, sourced from the live web. If the agent isn't confident about a field, it returns `NOT_FOUND` for that value.

curl -X POST 'https://api.olostep.com/v1/answers' \\
  -H 'Authorization: Bearer $OLOSTEP\_API\_KEY' \\
  -H 'Content-Type: application/json' \\
  -d '{
    "task": "What is Olostep and what does it do?",
    "json": {
      "company\_name": "",
      "description": "",
      "main\_use\_case": "",
      "website": ""
    }
  }'

Response:

{
  "id": "answer\_y0s1evxe8f",
  "object": "answer",
  "created": 1774369398,
  "metadata": {},
  "task": "What is Olostep and what does it do?",
  "result": {
    "json\_content": "{\\"company\_name\\":\\"Olostep\\",\\"description\\":\\"Olostep is a Web Data API that helps AI teams search, crawl, scrape and structure web data through a single, developer-friendly platform\\",\\"main\_use\_case\\":\\"Real-time extraction and structuring of web data for AI teams, data pipelines and automation\\",\\"website\\":\\"https://www.olostep.com/\\"}",
    "json\_hosted\_url": "https://olostep-storage.s3.us-east-1.amazonaws.com/answer\_y0s1evxe8f.json",
    "sources": \[
      "https://www.facebook.com/StartupPakistanSP/posts/olostep-launches-today-to-the-public-what-started-as-an-idea-in-italy-has-now-gr/1182671233896851/",
      "https://docs.olostep.com/get-started/welcome",
      "https://github.com/olostep",
      "https://medium.com/red-buffer/olostep-web-data-api-for-ai-and-research-automation-be8c93c28ef1",
      "https://www.olostep.com/pricing",
      "https://www.linkedin.com/company/olostep",
      "https://www.olostep.com/blog/about-olostep",
      "https://www.olostep.com/"
    \]
  }
}

Your requested answer, formatted according to the `json` parameter, is in `result.json_content`:

{
  "company\_name": "Olostep",
  "description": "Olostep is a Web Data API that helps AI teams search, crawl, scrape and structure web data through a single, developer-friendly platform",
  "main\_use\_case": "Real-time extraction and structuring of web data for AI teams, data pipelines and automation",
  "website": "https://www.olostep.com/"
}

> When you don't pass the `json` parameter, the API returns a JSON object with the answer text inside a `result` field. You can also pass a string describing the data you want instead of a JSON object.

* * *

## Searches

Search the web with a plain-English query and get deduplicated results with URLs, titles, and descriptions.

curl -X POST 'https://api.olostep.com/v1/searches' \\
  -H 'Authorization: Bearer $OLOSTEP\_API\_KEY' \\
  -H 'Content-Type: application/json' \\
  -d '{"query": "Olostep web scraping API for AI agents"}'

Response:

{
  "id": "search\_3v8hfo3yys",
  "object": "search",
  "created": 1774366508,
  "metadata": {},
  "query": "Olostep web scraping API for AI agents",
  "result": {
    "json\_hosted\_url": "https://olostep-storage.s3.us-east-1.amazonaws.com/search\_3v8hfo3yys.json",
    "links": \[
      {
        "url": "https://www.olostep.com/",
        "title": "Olostep - Web Data API for AI, Crawling & Data Extraction",
        "description": "Olostep is a Web Data API that helps AI teams search, crawl, scrape and structure web data through a single, developer-friendly platform. Built for modern AI ..."
      },
      {
        "url": "https://docs.olostep.com/get-started/welcome",
        "title": "Olostep Docs: Welcome to Olostep",
        "description": "Introduction. The Olostep API is the best web search, scraping and crawling API for AI used by some of the leading startups in the world."
      },
      {
        "url": "https://www.olostep.com/blog/best-web-scraping-tools",
        "title": "Best Web Scraping Tools: 11 Picks That Actually Scale | Olostep Blog",
        "description": "Best web scraping API for scalable, structured extraction. Olostep: Best for structured JSON, recurring batch workloads, and parser-driven ..."
      }
    \]
  }
}

### Search with Content Scraping

Chain Searches with Scrapes for a full AI research pipeline — search for relevant URLs, then pass each into `/v1/scrapes` to retrieve full page content.

import os
from olostep import Olostep

client \= Olostep(api\_key\=os.environ\["OLOSTEP\_API\_KEY"\])

\# Step 1: search
search \= client.searches.create(query\="Olostep web scraping API for AI agents")

\# Step 2: scrape each result for full content
for link in search.result.links\[:3\]:
    scrape \= client.scrapes.create(
        url\_to\_scrape\=link.url,
        formats\=\["markdown"\]
    )
    print(f"\\n\--- {link.title} ---")
    print(scrape.result.markdown\_content\[:300\])

* * *

## SDKs

Our SDKs provide a convenient way to interact with all Olostep features and automatically handle polling for async operations like crawls and batches.

### Python

pip install olostep

import os
from olostep import Olostep

client \= Olostep(api\_key\=os.environ\["OLOSTEP\_API\_KEY"\])

\# Scrape a URL
result \= client.scrapes.create(
    url\_to\_scrape\="https://example.com",
    formats\=\["markdown"\]
)
print(result.result.markdown\_content)

\# Search the web
search \= client.searches.create(query\="best LLM frameworks 2025")
for link in search.result.links:
    print(f"{link.title}: {link.url}")

\# Ask a question grounded on live web data
answer \= client.answers.create(
    task\="What is the latest version of Python?"
)
print(answer.result.json\_content)

\# Crawl a website (automatically polls for completion)
crawl \= client.crawls.create(
    start\_url\="https://docs.olostep.com",
    max\_pages\=50,
    include\_urls\=\["/features/\*\*"\]
)
for page in crawl.pages():
    content \= client.retrieve(retrieve\_id\=page.retrieve\_id, formats\=\["markdown"\])
    print(f"Crawled: {page.url}")

### JavaScript / TypeScript

npm install olostep

import Olostep from 'olostep';

const client \= new Olostep({ apiKey: process.env.OLOSTEP\_API\_KEY });

// Scrape a URL
const result \= await client.scrapes.create({
  url\_to\_scrape: 'https://example.com',
  formats: \['markdown'\]
});
console.log(result.result.markdown\_content);

// Search the web
const search \= await client.searches.create({
  query: 'best LLM frameworks 2025'
});
search.result.links.forEach(link \=> console.log(\`${link.title}: ${link.url}\`));

// Ask a question grounded on live web data
const answer \= await client.answers.create({
  task: 'What is the latest version of Python?'
});
console.log(answer.result.json\_content);

// Crawl a website (automatically polls for completion)
const crawl \= await client.crawls.create({
  start\_url: 'https://docs.olostep.com',
  max\_pages: 50,
  include\_urls: \['/features/\*\*'\]
});
for (const page of crawl.pages) {
  const content \= await client.retrieve({ retrieve\_id: page.retrieve\_id, formats: \['markdown'\] });
  console.log(\`Crawled: ${page.url}\`);
}

* * *

## Core Repositories

Repository

Description

**[olostep-js](https://github.com/olostep-api/olostep-js)**

Official JavaScript / TypeScript SDK

**[olostep-py](https://github.com/olostep-api/olostep-py)**

Official Python SDK

**[olostep-mcp-server](https://github.com/olostep/olostep-mcp-server)**

Olostep MCP Server

**[CLI](https://github.com/olostep-api/CLI)**

Olostep CLI

* * *

## Use Cases

-   **AI Agent Data Pipelines** — Feed LLMs and AI agents with fresh, structured web data
-   **Competitive Intelligence** — Monitor competitors, pricing, and market trends in real time
-   **Lead Generation** — Extract contact data, company info, and signals at scale
-   **Research Automation** — Automate web research workflows across thousands of pages
-   **Price Tracking** — Monitor e-commerce prices and product availability
-   **News & Content Aggregation** — Aggregate and structure content from across the web
-   **SEO & SERP Analysis** — Analyze search results and track rankings programmatically

* * *

## Integrations

**Agents & AI Tools**

-   🔨 [MCP Server](https://www.olostep.com/integrations/mcp-server) — Connect any MCP-compatible AI (Cursor, Claude, Windsurf) to live web data
-   🦜 [LangChain](https://www.olostep.com/integrations/langchain) — Add web scraping, search, and crawling to LangChain and LangGraph agents
-   🟣 [Apify](https://www.olostep.com/integrations/apify) — Deploy Olostep as a managed Apify Actor with scheduled runs and dataset exports
-   ⚡ [Mastra](https://www.olostep.com/integrations/mastra) — Integrate web scraping and search into Mastra AI agents and workflows

**Workflow Automation**

-   🔗 [Zapier](https://www.olostep.com/integrations/zapier) — Connect web data to 8,000+ apps without writing code
-   🔁 [n8n](https://www.olostep.com/integrations/n8n) — Use Olostep as native nodes in your self-hosted n8n workflows
-   🔀 [Relay](https://www.olostep.com/integrations/relay) — Add web scraping and search to Relay's workflow builder with human-in-the-loop support

[View all integrations →](https://www.olostep.com/integrations)

* * *

## Resources

-   📖 [Documentation](https://docs.olostep.com/get-started/welcome) — Full API reference and guides
-   🚀 [Get Started](https://www.olostep.com/) — Sign up and get your API key
-   💬 [Slack Community](https://olostep-users.slack.com/join/shared_invite/zt-2bfddyi8h-JzfjOgavg~98DJ1om1B5Lg#/shared-invite/email) — Get help and share what you're building
-   𝕏 [Twitter / X](https://x.com/olostep) — Follow for updates

* * *

**[olostep.com](https://www.olostep.com/)** · The web scraping, crawling, and search API powering the world's leading AI agents and companies.