Report a crawler request
Send one crawler request from your server for bot and AI crawler tracking.
POST https://usedatasense.com/api/crawls
No token: the website ID identifies the site, and Datasense verifies the crawler itself. Send it from your server, in the background, with a short timeout. Ready-made middleware is in bot traffic tracking.
sh
curl -X POST https://usedatasense.com/api/crawls \
-H "Content-Type: application/json" \
-d '{
"websiteId": "YOUR_SITE_ID",
"href": "https://example.com/docs/get-started",
"ai": { "userAgent": "Mozilla/5.0 … GPTBot/1.2", "ip": "20.171.207.1", "statusCode": 200 }
}'| Field | Required | Notes |
|---|---|---|
websiteId |
Yes | Your site ID. |
href |
Yes | The URL requested, without its query string. |
ai.userAgent |
Yes | The request's full User-Agent. |
ai.ip |
Recommended | The crawler's IP, checked against published ranges. |
ai.statusCode |
No | Your response status, to find pages crawlers get 404s on. |
ai.source |
No | A label for the reporter, such as nextjs. |
Answers {"success": true, "tracked": true} for a known crawler and "tracked": false otherwise. Reports for websites without bot tracking on their plan are accepted and dropped.