AI Crawlers

Bots deployed by AI companies to index web content for their LLMs. Major crawlers include GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot (Perplexity), Google-Extended (Google), Applebot-Extended (Apple), and ChatGPT-User (OpenAI browsing). Managing these crawlers via robots.txt and llms.txt is essential for both defense and optimization.