Skip to main content
Crawler analytics in Trackbase shows which crawlers visit your website, which pages they request and whether those requests succeed. A page that AI crawlers cannot read cannot be cited, which makes this the first thing to check when a page stays invisible. You find it under Crawlers in the menu. Crawlers do not run JavaScript the way browsers do, so a tracking script in the page does not see them. Trackbase records crawler requests server-side instead.

​
Three ways to connect

You set up each method under Settings, in the crawler analytics card. Until a connection exists, the Crawlers page shows a setup prompt. After connecting, it waits for the first data.

​
Which crawlers are recognized

Trackbase recognizes a crawler by its user agent and sorts it into a category: Unknown bots are grouped under a general label.

​
What the page shows

The Crawler analytics page in Trackbase with total requests, unique crawlers, top crawler, the crawl status bar and daily crawler requests by category. Example data.
  • Key numbers for the selected period: total requests, unique crawlers, the top crawler and today.
  • Crawl status: the share of crawler requests that succeeded or failed.
  • Timeline of crawler requests, with an optional 7-day moving average.
  • Breakdown per crawler and per category.
  • Top pages: the pages crawlers request most, with a status that shows whether the page is crawlable or how often requests fail.
  • Crawler request log: individual requests with date, crawler, page, status code and referrer. Switch it to failures only to debug a page.
The request log is based on the last 30 days.

​
What to look for

  • Important pages with no AI crawler requests. Check your robots.txt, your firewall and your CDN settings.
  • Pages with a failure status. Open the request log, filter on failures and look at the status codes.
  • Crawled but never cited. The crawlers can read the page, so the work is in the content. Follow the page in the Page tracker.
Last modified on November 12, 2026