Cloudflare Worker

Deploy a Cloudflare Worker that intercepts AI bot visits and sends them to Robot Speed. Works with Cloudflare's free plan.

Prerequisites

  • βœ“A Cloudflare account (free)
  • βœ“Your domain on Cloudflare with proxy enabled (orange cloud)
  • βœ“Your Robot Speed API key (Settings β†’ API in your dashboard)
1

Create a Worker

  1. Go to your Cloudflare Dashboard
  2. Click Workers & Pages in the sidebar
  3. Click Create Worker
  4. Give it a name (e.g. ai-bot-tracker)
2

Paste the code

Replace the default code with the code below, then click Deploy.

export default {
  async fetch(request, env, ctx) {
    // Forward the request to your origin β€” transparent proxy
    const response = await fetch(request);

    // Detect AI bots from user-agent
    const ua = request.headers.get("user-agent") || "";
    const bots = [
      ["ChatGPT-User", "openai"],
      ["OAI-SearchBot", "openai"],
      ["PerplexityBot", "perplexity"],
      ["Perplexity-User", "perplexity"],
      ["Claude-User", "anthropic"],
      ["Claude-SearchBot", "anthropic"],
      ["Google-Extended", "google"],
      ["Google-Agent", "google"],
      ["MistralAI-User", "mistral"],
      ["xAI-Grok", "xai"],
      ["GrokBot", "xai"],
      ["Grok-DeepSearch", "xai"],
      ["DeepSeekBot", "deepseek"],
      ["cohere-ai", "cohere"],
      ["Applebot-Extended", "apple"],
    ];

    const bot = bots.find(([sig]) => ua.includes(sig));

    if (bot) {
      const url = new URL(request.url);
      const path = url.pathname;

      // Skip noise: feeds, tags, static files
      if (!/\/(feed|tag|category)\b/i.test(path) && !/\.\w{2,4}$/.test(path)) {
        // Fire-and-forget β€” never blocks the response
        ctx.waitUntil(
          fetch(env.ANALYTICS_ENDPOINT, {
            method: "POST",
            headers: {
              "Content-Type": "application/json",
              "X-API-Key": env.ANALYTICS_KEY,
            },
            body: JSON.stringify({
              bot_name: bot[0],
              bot_provider: bot[1],
              url_path: path,
              user_agent: ua.slice(0, 500),
              host: request.headers.get("host")?.replace(/^www\\./, ""),
              source: "cloudflare_worker",
            }),
          }).catch(() => {})
        );
      }
    }

    // Return the original response β€” zero latency impact
    return response;
  },
};
3

Configure variables

Go to Worker Settings β†’ Variables, and add:

ANALYTICS_ENDPOINT
https://www.robot-speed.com/api/ai-bot-visits
ANALYTICS_KEY

Your Robot Speed API key β€” find it in your dashboard: Settings β†’ API

4

Add the route

  1. In the Worker, go to Triggers β†’ Add Route
  2. Route: yourdomain.com/*
  3. Select your zone (domain)
5

Verify DNS proxy

In DNS β†’ Records, make sure your domain has the orange cloud (Proxied) enabled. If the cloud is grey (DNS only), the Worker won't see the traffic.

Verification

Test with this command:

curl -H "User-Agent: Mozilla/5.0 (compatible; ChatGPT-User/1.0)" https://yourdomain.com/

A few seconds later, you should see the mention in your Robot Speed dashboard.