> ## Documentation Index
> Fetch the complete documentation index at: https://academy.trackbase.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# See which AI crawlers read your website

> Crawler analytics shows which AI crawlers and search bots request which pages of your website, recorded server-side. Learn how to set it up.

Crawler analytics in Trackbase shows which crawlers visit your website, which pages they request and whether those requests succeed. A page that AI crawlers cannot read cannot be cited, which makes this the first thing to check when a page stays invisible. You find it under **Crawlers** in the menu.

Crawlers do not run JavaScript the way browsers do, so a tracking script in the page does not see them. Trackbase records crawler requests server-side instead.

## Three ways to connect

| Method | When to use it |
| - | - |
| [WordPress plugin](https://academy.trackbase.ai/wordpress-plugin.md) | Your website runs on WordPress. |
| [Cloudflare Workers](https://academy.trackbase.ai/cloudflare.md) | Your website is behind Cloudflare, on any Cloudflare plan. |
| [Cloudflare Logpush](https://academy.trackbase.ai/cloudflare.md) | Your website is behind Cloudflare on an Enterprise plan. |

You set up each method under **Settings**, in the crawler analytics card. Until a connection exists, the **Crawlers** page shows a setup prompt. After connecting, it waits for the first data.

## Which crawlers are recognized

Trackbase recognizes a crawler by its user agent and sorts it into a category:

| Category | Examples |
| - | - |
| AI crawlers | GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, DeepSeekBot |
| Search engines | Googlebot, Bingbot, DuckDuckBot, YandexBot |
| Social | LinkedInBot, FacebookExternalHit, Slackbot |
| SEO tools | AhrefsBot, SemrushBot, Screaming Frog |
| Monitoring | UptimeRobot, Pingdom |

Unknown bots are grouped under a general label.

## What the page shows

![The Crawler analytics page in Trackbase with total requests, unique crawlers, top crawler, the crawl status bar and daily crawler requests by category. Example data.](https://mintcdn.com/trackbase/3L0jUS0Gnf-g9E7B/images/crawlers.png?fit=max\&auto=format\&n=3L0jUS0Gnf-g9E7B\&q=85\&s=b5782b7a2dcda20cdf27d3e269e8e219)

* **Key numbers** for the selected period: total requests, unique crawlers, the top crawler and today.
* **Crawl status**: the share of crawler requests that succeeded or failed.
* **Timeline** of crawler requests, with an optional 7-day moving average.
* **Breakdown per crawler** and **per category**.
* **Top pages**: the pages crawlers request most, with a status that shows whether the page is crawlable or how often requests fail.
* **Crawler request log**: individual requests with date, crawler, page, status code and referrer. Switch it to failures only to debug a page.

The request log is based on the last 30 days.

## What to look for

* **Important pages with no AI crawler requests.** Check your robots.txt, your firewall and your CDN settings.
* **Pages with a failure status.** Open the request log, filter on failures and look at the status codes.
* **Crawled but never cited.** The crawlers can read the page, so the work is in the content. Follow the page in the [Page tracker](https://academy.trackbase.ai/page-tracker.md).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.