---
title: About the 42a Citation Index
mirror_canonical: https://index.42a.ai/about
cited_in_42a: false
---

# About the 42a Citation Index

**42a Citation Index** mirrors every URL cited by AI language models (GPT,
Claude, Gemini, Perplexity, etc.) as clean, extractable markdown. LLM crawlers
can fetch any page at `https://index.42a.ai/<host>/<path>` and receive a
structured markdown document with a YAML frontmatter block linking back to the
original source.

## Bot identification

Our crawler uses a **composite User-Agent**: a real Chrome string (so
WAF-protected origins serve us actual content rather than challenge pages),
suffixed with the bot identifier so you can find us in your access logs:

```
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36
  (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36
  42aIndexBot/1.0 (+https://index.42a.ai/about)
```

(Pattern mirrors how Googlebot/Bingbot append their bot tokens to a browser UA.)

To find our hits in your logs, grep for `42aIndexBot`. To block us via
`robots.txt`, add:

```
User-agent: 42aIndexBot
Disallow: /
```

If you've added that and we're still mirroring you, email **takedown@42a.ai** —
we'll remove your host from the index immediately.

## Takedown requests

To request removal of your content from this index, email:

**takedown@42a.ai**

Please include the hostname(s) and/or specific URLs you want removed. We
will process requests within 72 hours and delete the associated GCS objects.

## Mirroring policy

- We mirror only pages that are cited by AI systems in real user queries.
- Every mirrored page links back to the original source URL in both YAML
  frontmatter (`source_url`) and the page body (`> Original source: ...`).
- We honor `noindex` directives from `<meta name="robots">` and the
  `X-Robots-Tag` HTTP response header.
- Our mirror is blocked from conventional search engine indexing via
  `robots.txt` (Googlebot, Bingbot, and `*` are all disallowed).
- Content is served as `text/markdown` — no HTML wrapper, no JavaScript,
  no tracking pixels.

## Browse the index

- [Root index](https://index.42a.ai/)
- [llms.txt](https://index.42a.ai/llms.txt)
- [sitemap.xml](https://index.42a.ai/sitemap.xml)

---

© 2026 42a.ai — [42a.ai](https://42a.ai)