LeaderboardEvery site, rankedFree toolsChecks, no sign-upAgentsYour agent fixes itDocsThe API and MCP

AI crawler checker

Which AI crawlers a site’s robots.txt lets in, from GPTBot and ClaudeBot to PerplexityBot, and whether it has an llms.txt.

More checksPagesDomain and email

The answer for allbirds.com

allbirds.comEvery AI crawler checked may read this site
  • GPTBotOpenAIAllowed
  • OAI-SearchBotOpenAIAllowed
  • ChatGPT-UserOpenAIAllowed
  • ClaudeBotAnthropicAllowed
  • Claude-SearchBotAnthropicAllowed
  • PerplexityBotPerplexityAllowed
  • Google-ExtendedGoogleAllowed
  • bingbotMicrosoftAllowed
It has an llms.txt, a plain map of the site for assistants.

Read from the site’s robots.txt as each crawler would read it, by the name it obeys. A site can also turn crawlers away at its firewall, which robots.txt does not show; clickd’s full read checks that too.

That was one page of allbirds.com.clickd reads its first 100 pages against one rulebook and ranks every fix by what it is worth. Free, about a minute.
Read the whole site

Being cited by assistants

Its line of workOnline shopsA best guess from its home page

What people ask assistants in this line of work

  • Where can I buy a cast iron pan made in the US, under $60?
  • Which running shoes are best for flat feet, and where can I buy them online?
  • Where can I buy a gift that ships in two days for under $50?

Examples of the work, not this site’s own searches.

What its robots.txt settles

  • Every AI crawler checked may read it, so whether an assistant cites it comes down to what its pages say.

Facts from its robots.txt, not estimates.

What sites in this line of work miss most

  • Add structured data
    64%
  • Remove ratings with no visible reviews
    9%
  • Describe images
    85%

The share failing each rule across the 3,512 shops sites clickd has read, not on this site. A read of allbirds.com says whether it is one of them.

A citation reportExample

Not allbirds.com’s figures: it has not been asked yet. A report puts a site’s own searches to the assistants people use, and code reads which sites each answer cites.

  1. Where can I buy a cast iron pan made in the US, under $60?Google AI OverviewsNot citedChatGPT searchNot citedPerplexityCited, 2nd
  2. Which running shoes are best for flat feet, and where can I buy them online?Google AI OverviewsNo answer shownChatGPT searchCited, 4thPerplexityNot cited
  3. Where can I buy a gift that ships in two days for under $50?Google AI OverviewsNot citedChatGPT searchNamed, no linkPerplexityNot cited

Example. In this example the site is cited in 2 of 8 answers and named without a link in 1, and Google showed no AI Overview for 1 search. Answers change from one day, place and person to the next, so a real report is a sample, not a ranking.

Measured, not guessedclickd asks a site’s own searches of the assistants people use, reads which sites each answer cites, and hands the fixes to your agent, checked on the live site.
How clickd gets a site cited

What does it check?

Rules from the rulebook, and how many of the sites clickd has read miss each.

  1. Serve a robots.txtServe a plain text /robots.txt.
    16%of sites miss it
  2. Let AI crawlers inRemove the robots.txt rules that shut out AI crawlers. On Cloudflare, also turn off "Block AI bots", which writes these rules for you.
    5%of sites miss it
  3. Stop refusing AI crawlersStop the firewall or CDN from refusing AI crawlers. This test only imitates their names, so confirm in the CDN dashboard.
  4. Add an llms.txtAdd /llms.txt: a short plain text map of the site for assistants. Its effect is unproven, so treat it as cheap insurance.
    79%of sites miss it
A robots.txt open to assistantsExample

example.com/robots.txt

# Search and answer engines that cite their sources
User-agent: OAI-SearchBot
User-agent: PerplexityBot
Allow: /

# Crawlers that read for training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Allow: /

User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

Each AI crawler is named on its own line, and nothing under its name or under * shuts it out of the pages meant to be found.

Questions

Which AI crawlers are there?

GPTBot and OAI-SearchBot for OpenAI, ClaudeBot for Anthropic, PerplexityBot, Google-Extended, Applebot-Extended, and more. Each is named in robots.txt on its own.

Does blocking GPTBot keep my site out of ChatGPT?

It keeps OpenAI from reading pages to train on. ChatGPT’s search reads under OAI-SearchBot, a separate name with its own line in robots.txt.

Does anything read llms.txt?

Some agents and tools do, when asked to read a site. No major search engine has said it uses one.

The whole site

One page is one part. A read checks every page against the whole rulebook and says what to fix first.

See allbirds.com’s report