AI Crawlers

Which AI engines read your site, page by page

Refinea reads your site's logs and shows you when GPTBot, ClaudeBot, PerplexityBot and other AI crawlers open your pages. For each visit you see the purpose, from answering a person to training a model, and next to it the people who arrive from AI assistants.

Want to see how AI engines see you today first? Get the free analysis.

  • ChatGPT
  • Gemini
  • Claude
  • Google AI Mode
  • AI Overview
  • Perplexity
  • Copilot
  • DeepSeek
  • Grok
  • Mistral
Crawler Analytics: AI crawler visits split into AI Citations, AI Training and AI Indexing, day by day
Refined Marketers
  • Caffeina
  • AIRC
  • Finbooks
  • WeHunt
  • GECOS
  • Grenke
  • Garmont Tactical
  • StoryWalking
  • Omni
  • Seoleader
  • Fabio Antichi
  • Axyon AI
  • Unflat
Named in thesurvey of the tools Italian SEO experts use (2026)
Pre-accelerated byB4i·Participant in TEF Ignition byTech Europe Foundation·Finalist atStartCup Lombardia

AI Citations

Crawlers that open a page to answer a person while they're asking the question.

AI Indexing

Crawlers that build the engines' search index.

AI Training

Crawlers that take content to train a model.

Human Visits

People who click from ChatGPT or Perplexity and land on your site.

What you see

Every AI crawler visit, with who made it and why.

Platform by platform

OpenAI, Anthropic, Perplexity, Google and the others: how many visits, from which crawler and when you last saw it.

The AI platforms that visited the site: Anthropic, OpenAI, ByteDance, Meta, Amazon, Google, Microsoft, Perplexity and DuckDuckGo, with their crawlers
Only real crawlers

Every visit is checked against the identity the platform publishes. You spot anything posing as GPTBot right away.

Requests to the site as Crawler Analytics reads them: each crawler with its verification check and the purpose of the visit
Page by page

For each page you see how many crawlers opened it for citations, for the index and for training, and how many people AI brought to it.

The site's pages with visits for AI Citations, AI Training and AI Indexing, and the people who arrived from AI
Logs in plain view

One line per request: time, crawler, page, server response. Search for a page and find out who read it.

The logs line by line: Meta, Google, OpenAI and Amazon, with page, response code and time
From assistant to page

Which AI assistant people come from and which page they land on.

The path from platforms to pages: ChatGPT, Claude and Gemini linked to the pages where people land
With Google Analytics

Connect GA4 and see the sessions that come from AI assistants and how many of them are engaged.

The Google Analytics connection for measuring traffic from AI assistants
The method

Answers show what AI writes. Logs show what it reads.

Monitoring shows you how engines answer your questions. Crawler Analytics shows you whether they open your pages to build those answers, and Google Analytics shows how many people then reach your site.

  • What they answerEngine answers to the questions in your market, read every day.
  • What they readAI crawler visits, page by page, with the purpose of each visit.
  • Who arrivesPeople who click from an AI assistant and land on your site.

Explore AI Monitoring The method in detail

The pages read by AI crawlers, with the purpose of each visit
What to do

When AI crawlers can't find their way, you get an Action.

A robots.txt that blocks GPTBot or ClaudeBot, or doesn't point to the sitemap, becomes an Action with the questions at stake. The technical SEO agents prepare the work, and your team approves before anything changes.

  • Robots.txt & LLMs.txt AuditorChecks that AI engines can find, read and cite the site.
  • llms.txtPrepares the site index for AI engines and agents.
  • Structured Data GeneratorWrites JSON-LD that's ready to paste.

Explore Actions Explore agents

Nike's site Actions: structured data, FAQPage markup and the llms.txt file, with the impact of each
Getting started

Your first crawler shows up within minutes.

  1. 1

    Pick how your site is served

    Cloudflare, WordPress, Vercel or Webflow.

  2. 2

    Follow the guide

    Generate the token and complete the steps shown, one at a time.

  3. 3

    See who shows up

    The first visits appear within minutes. If they stop coming in, Refinea lets you know.

Is your site served some other way? Tell us how and we'll work it out together.

The Crawler Analytics connection: Cloudflare, WordPress, Vercel and Webflow
“With Refinea we’re making WeHunt’s expertise more visible in AI answers too. Over the work we’ve done together, visits from AI assistants have grown by 121%.”
Davide FagnaniMarketing at WeHuntWeHunt

Put agents to work on everything AI reads about you.

With Pro, we set up Refinea with you on day one. Agents prepare the work, and your team approves it before anything goes out.

Want to see how AI engines see you today first? Get the free analysis.

Frequently asked questions about AI crawlers

How do you read my site's logs?

Through a connection on Cloudflare, WordPress, Vercel or Webflow: pick yours, generate the token and follow the guide. The first visits appear within minutes.

My site is served some other way. Can it still work?

Tell us how your site is served and we'll work it out together.

What's the difference between AI Citations, AI Indexing and AI Training?

AI Citations covers the crawlers that open a page to answer a person in that moment. AI Indexing covers the ones that build the search index. AI Training covers the ones that take content to train a model.

How do you know a crawler is really GPTBot?

Every visit is checked against the identity the platform publishes, through a reverse DNS check or a check against its IP address ranges. Visits that fail the check are marked as unverified.

Can I also see the people who come from AI?

Yes, in two ways: from the logs, for people who click from ChatGPT or Perplexity, and from Google Analytics, if you connect it, with sessions and engaged sessions for each assistant.

What happens if the connection stops working?

If no visits come in for a few days, Refinea lets you know and tells you what to check.