Question 1

What is llms.txt and do I actually need one?

Accepted Answer

llms.txt is a plain-text file at your site's root, like robots.txt -- but instead of controlling access, it gives AI assistants a short, structured summary of what your site is and links to its most important pages. It's optional: AI systems can read your site without one. It just removes the guesswork, which can make the summary they give of you more accurate.

Question 2

What's the difference between llms.txt and llms-full.txt?

Accepted Answer

llms.txt is meant to stay short -- a description plus a curated list of links. llms-full.txt takes the same idea further: one file with your full content inlined, so an AI system gets everything in a single fetch instead of following links. Most sites only need llms.txt; llms-full.txt tends to make more sense for smaller sites or documentation sets.

Question 3

What's the difference between robots.txt and llms.txt?

Accepted Answer

robots.txt controls access: it tells crawlers what they're allowed to fetch, and well-behaved bots respect it. llms.txt doesn't control anything -- it's just context for whichever crawlers choose to read it. You can have either file without the other; they solve different problems.

Question 4

Which of these crawlers should I actually allow?

Accepted Answer

Depends what you want. GPTBot trains OpenAI's models, ClaudeBot trains Claude, and bots like OAI-SearchBot or Perplexity-User power live answers instead of training -- each maps to a different product. Allow the one behind whatever you want your content trainable in or citable by; blocking one doesn't affect the others.

Question 5

Does blocking these crawlers hurt my regular Google ranking?

Accepted Answer

No. The crawler that builds normal Google Search results is Googlebot, and it isn't on this list -- it's separate and allowed by default. Google-Extended, which we do check, only feeds Gemini and AI Overviews training. Blocking it opts you out of that without touching your regular search rankings.

Question 6

I found some of these blocked in my robots.txt -- was that on purpose?

Accepted Answer

Often not. Plenty of sites are running a robots.txt written before most of these bots existed, or a boilerplate rule copied in for something else and never revisited. Worth checking either way: deliberately blocking AI crawlers to keep content out of model training is a legitimate choice too, not necessarily a mistake.

Question 7

If I allow everyone and add llms.txt, will AI actually cite me?

Accepted Answer

Not guaranteed -- this removes barriers, not competition. Whether ChatGPT, Perplexity, or Google's AI Overviews cite you still comes down to whether your content answers the question better than whatever else they found.

Question 8

How often should I re-run this check?

Accepted Answer

Whenever you touch your robots.txt, and every few months regardless. This list already covers 14 crawlers across 9 companies, and new ones keep launching -- a rule you wrote a year ago may be silently blocking something that didn't exist yet.

Can AI see
your site?

Why this matters

Can AI seeyour site?

Why this matters

Can AI see
your site?