Claude, Gemini and the Rest

ChatGPT, Perplexity and Google Search get most of the attention. Behind them sit four more vendors with their own crawlers and their own retrieval paths: Anthropic, Google's Gemini products, Meta and Microsoft. Each one answers to a different control, and most sites handle all of them with one block of robots.txt copied from a "block AI" list.

That block is how retrieval gets cut off by accident. A rule meant to keep content out of training also removes the site from a search crawler, and nobody notices, because no dashboard reports it.

This lesson goes vendor by vendor: what each agent does, what each vendor says about it, and where the vendor says nothing. The win is an access matrix for your own site, one row per agent.

Claude: three agents, and a search Claude chooses to run

Anthropic documents three agents (support.claude.com, updated 2026-04-07):

Agent	Role	What blocking it does
ClaudeBot	Training crawls	Keeps your content out of Anthropic's training crawls. Search and user fetches run as separate agents.
Claude-SearchBot	Improves search result quality	Likely keeps your pages out of the results Claude search draws on. This is our reading: Anthropic does not state the consequence outright.
Claude-User	Fetches a page when a user asks	Claude cannot fetch your pages live for that user. Also our reading of Anthropic's description.

Anthropic adds one warning worth acting on: blocking its IP addresses "impedes our ability to read your robots.txt file." A firewall rule can stop Claude's agents from even learning what you allow. Rendering matters here too. Vercel's crawler study (published December 2024) found Claude's crawler fetched JavaScript files on 23.84% of requests without executing them, and independent re-testing in June 2026 found ClaudeBot and Claude-SearchBot still do not render JavaScript. Put your facts in the raw HTML.

Retrieval is Claude's call, made per query, and others steer it. A company that builds on Claude writes a system prompt that shapes when it searches. In the Claude app, search is a user toggle with no published default. The API's organization-level setting defaults to on, which says nothing about what the consumer app does. Without visible citations, you do not know which mode answered.

How Claude cites when it searches

Anthropic's web search tool returns each result with a URL, a title and a page_age field, and Claude cites text spans: a quoted passage of up to 150 characters, not a whole page. A May 2026 Muck Rack study of more than 25 million links found Claude cited sources in about 55% of the responses it examined, the most selective of the engines it measured. That figure came from our first research pass, not the verified one, and it is not a vendor number, so treat it as a rough guide: many Claude answers carry no link at all.

Like OpenAI and Perplexity, Anthropic's API returns the full set of results a search retrieved separately from the passages Claude cited. A page can be read and not quoted. The citation is the visible part of what Claude used, not a full record of it.

page_age is the kind of field that gets turned into advice. It is exposed, not shown to be weighted. No vendor says page age influences which sources Claude selects, so do not bump dates to chase it.

Now the measurement point. 99Visibility asks Claude without web search, so our Claude column shows what the model already knows: Stage 0, parametric memory frozen at a cutoff. That is our measurement choice, not a property of Claude, and it is why that column rarely shows a citation. It tells you what Claude believes about you when nobody hands it the web. If it says Ledgerly costs $39, that is a stale belief, and no access fix will change it until a newer model ships. The Visibility Score weights each platform by market share in its coverage multiplier (ChatGPT 40%, Google AI Overviews 20%, Perplexity 20%, Gemini 10% and Claude 10%), so read the Claude column for what it says, not for how far it moves the total.

Gemini: one brand, four Google paths

"Gemini" is not one place your content can appear. Google runs four generative retrieval paths: AI Overviews, AI Mode, the Gemini apps, and Grounding with Google Search in the Gemini API. They answer to different controls.

• Google-Extended is a robots.txt token with no crawler of its own. Disallowing it removes your content from training and grounding for the Gemini apps and Vertex AI. Google states it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." So blocking Google-Extended keeps you out of Gemini app answers and leaves you in AI Overviews, the opposite of what many site owners expect.
• Googlebot controls AI Overviews and AI Mode, through ordinary Search eligibility. That path is covered in Optimizing for Google AI Overviews.
• Grounding with Google Search in the Gemini API is a separate path that a developer switches on and pays for. The model "automatically generates one or multiple search queries" and runs them, and Google bills grounding separately from tokens. A Gemini answer inside someone else's product may be grounded or may not, depending on how that developer built it.
• Google-CloudVertexBot crawls only when a site owner asks it to, and Google says it has "no effect on Google Search or other products."

Googlebot renders JavaScript, so client-side content is less of a problem on Google's paths than anywhere else. 99Visibility's Gemini column asks the Gemini API with Google Search grounding offered, so the model decides per query whether to ground its answer. That is the API, not the Gemini app. AI Overviews is a separate column, read from the live Google results page. Google says its surfaces "may use different models and techniques," so expect the two columns to disagree, and do not treat either one as a stand-in for the other.

Meta AI: the one vendor that ties access to citation

Across every vendor document in our evidence base, one sentence states that letting a crawler in helps you get cited. It is Meta's: "Allowing Meta-WebIndexer in your robots.txt file helps us cite and link to your content in Meta AI's responses." (developers.facebook.com, 2026-05-21)

Meta's other agents are less clear. Meta-ExternalAgent is documented for training AI models and indexing content, so blocking it likely affects both, though Meta does not spell out the consequence. Meta-ExternalFetcher is documented as able to bypass robots.txt, much like Perplexity-User.

Be honest about what you cannot see. When our evidence review ran in August 2026, no GEO vendor tracked Meta AI, and 99Visibility does not query it either. The defensible move is cheap: allow Meta-WebIndexer, confirm your CDN lets it through, and accept that you cannot measure the effect today. The Crawler Access check probes Meta-WebIndexer alongside the other agents, so a CDN refusal shows up there.

Microsoft Copilot: the Bing index

Microsoft documents the Bing index as the backing store for Copilot, AI-generated summaries in Bing, and Grounding with Bing Search for developers. Bingbot is the control: a page missing from Bing's index is missing from the index Copilot draws on. Verify your site in Bing Webmaster Tools and confirm your key pages are indexed. It is free, and your Google work does not cover it.

Two Microsoft-specific details:

• data-nosnippet works on Bing. Since October 15, 2025, content marked with it "is still indexed normally, but it will be excluded from snippets and AI summaries." It hides display, not ranking, so an accidental wrapper hides you from Copilot summaries.
• Bing Webmaster Tools shows grounding queries. Its AI Performance report lists "the key phrases the AI used when retrieving content that was referenced in AI-generated answers" across Copilot, Bing AI summaries and select partners. It is the only surface anywhere that shows the queries an AI actually ran, not prompts someone guessed. It has no API, the grounding-query data is a sample, and Microsoft calls its Citation Share "an observational metric – not a ranking system or a competitive scoreboard."

IndexNow tells Bing about changes quickly. Microsoft asserts it helps AI systems reference the current version of a page, but the specification never mentions AI and Google is not among the engines that consume it. Treat it as free Bing hygiene with no AI guarantee. 99Visibility does not query Copilot.

The rest of the roster

A few more agents appear in most robots.txt files. Here is what each vendor documents, and where the consequence is our inference:

Agent	Vendor	Role	What blocking it does
Applebot-Extended	Apple	Training	Excludes content from Apple foundation model training. Apple says the token does not crawl and does not affect Applebot, which serves Siri, Spotlight and Safari.
Amazonbot	Amazon	Training and retrieval	Amazon says the content improves its services, including Alexa answers, so blocking removes you from that source.
DuckAssistBot	DuckDuckGo	Retrieval	DuckDuckGo says it fetches pages for DuckAssist answers, so blocking keeps you out of them.
CCBot	Common Crawl	Training	Removes you from a corpus many model trainers use. Which models that affects is not documented (our inference).
Bytespider	ByteDance	Training	No published consequence, and no retrieval surface to lose (our inference).

One principle covers every row. A training opt-out is a business decision: no vendor documents a visibility effect either way, and nobody has measured one. A retrieval block is a visibility loss you can name. Decide each row on purpose.

Common mistakes

• Copying a "block AI" list. Those lists mix training and retrieval agents. Check each name against its vendor's description.
• Blocking Google-Extended to leave AI Overviews. It controls the Gemini apps and Vertex AI, not Search.
• Reading our Claude column as Claude search. 99Visibility asks Claude without web search. A wrong answer there is a stale belief, not an access failure.
• Bumping dates for page_age. The field is exposed. Nobody has shown it is weighted.
• Forgetting Meta-WebIndexer and Bingbot. They often have no group of their own, so a broad User-agent: * rule decides their fate.
• Blocking by IP and expecting robots.txt to work. Anthropic warns IP blocks stop it from reading your robots.txt at all.
• Assuming Google covers Copilot. Copilot draws on Bing's index. Check Bing Webmaster Tools on its own.