Crawl Budget

The number of pages an AI crawler will index from your domain within a given timeframe. AI crawlers prioritize pages based on perceived importance. Optimizing crawl budget means ensuring llms.txt guides crawlers to important pages, using clean internal linking, and removing low-value pages from crawl scope.