AI crawlers are the programs OpenAI, Anthropic, Perplexity, Google and others use to read the web. You recognise them by the name they declare in their requests, the user agent, and they leave traces in your server logs.
Three different jobs
They don’t all do the same thing. Some collect text to train models, such as GPTBot and ClaudeBot. Others build the index the engine draws sources from, such as OAI-SearchBot and Claude-SearchBot. Others open a page at the moment a user asks a question, such as ChatGPT-User, Claude-User and Perplexity-User. Blocking a training bot doesn’t remove your site from answers; blocking the search and answer bots does.
How to manage them
The robots.txt file states which bots may read which pages. Google is a special case: Google-Extended covers Gemini training and doesn’t affect inclusion in Search, AI Overviews included. Before blocking anything, look at the logs: which bots arrive, what they read and how many people then arrive from AI assistants.
Refinea reads your site’s logs and shows AI crawler visits page by page: see AI crawlers.