See which pages AI crawlers read

If assistants never fetch your pricing page, they can't quote your prices. If their crawler hits 404s, they learn your site is broken. Crawler traffic shows what AI models know about you and what they're missing.

Analytics scripts run in the browser, and crawlers don't execute them, so script-based tools filter bots out instead of reporting them.

On Pro and Team, a few lines of middleware on your server report each crawler request. Datasense names the crawler and its company, checks its IP against published ranges to catch spoofed user agents, and sorts requests into AI answers, indexing, training and other bots.

1. 1Add the bot tracking middleware for your stack (Next.js, Express, Cloudflare or any backend) from Settings → Bot traffic.
2. 2Open the Bot traffic page for requests over time by category, provider and crawler.
3. 3Check the pages crawlers read most, the discovery files they fetch (robots.txt, llms.txt, sitemaps) and the pages that returned 404.

• Fix pages that return 404 to crawlers; redirect old URLs.
• If crawlers never reach a key page, link it from pages they do read and add it to your sitemap.
• Publish an llms.txt and Markdown versions of your docs, then watch crawlers pick them up.

• Bot tracking is on Pro and Team, and needs a few lines on your server: crawlers never run the browser script.