Rendering, noindex and Snippet Controls

Your robots.txt allows the right bots. Your firewall lets them in. The crawler fetches your pricing page and gets a 200. It can still leave with nothing.

Two things cause that. The page's facts may exist only after JavaScript runs, and the AI crawlers do not run JavaScript. Or the page may carry a directive, often inherited from a template nobody remembers editing, that tells engines not to index it or not to quote it. Both failures are silent, and the page looks perfect in your browser.

This lesson finishes Stage 1, access. It also holds the strongest single piece of evidence in the field: the rendering gap.

The AI crawlers read HTML, not your app

When a browser loads a client-rendered page, it downloads a thin HTML shell, runs the JavaScript, calls your APIs and builds the page you see. A crawler that does not execute JavaScript stops at the shell. Whatever your scripts would have added, it never receives.

Weigh that source properly. It is a one-month snapshot from December 2024, measured on nextjs.org, a sample skewed toward Next.js sites. Microsoft Copilot was left out because it has no unique user-agent. And Vercel sells server-side rendering, so it has an interest in the answer. The finding stands anyway, because it replicates: independent testing in June 2026 found that none of GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Meta-ExternalAgent or Bytespider rendered JavaScript.

Two crawlers are the exception. Googlebot renders JavaScript, and AppleBot renders through a browser-based crawler. So a client-rendered page can look healthy in Google and be empty to ChatGPT, Claude and Perplexity. "It ranks fine in Google" proves nothing here.

One nuance from Vercel: data shipped inside the initial HTML response, such as a JSON or React Server Components payload, may still be indexed. Data your page fetches after it loads is not there to index.

Be exact about what this lever does. Server rendering does not make an engine more likely to pick you. It decides whether your content exists in the candidate set at all. It is hygiene, not optimization, and it is the highest-confidence access fix in our evidence base.

Run the hydration check by hand

The 99Visibility Crawler Access check does not run a rendering or hydration diff, so this one is manual. It takes about five minutes a page. Start with the pages that state your facts: pricing, product, about and comparison pages.

Step 1: compare view-source with the rendered DOM

Open the page in Chrome and press Ctrl+U (Cmd+Option+U on a Mac). View-source shows the raw HTML the server sent, which is what a non-rendering crawler gets. Then open DevTools with F12 and look at the Elements panel: that is the DOM after JavaScript has run. Search both for the facts that matter. If a price, a plan name, a feature or a headline appears in Elements and not in view-source, the AI crawlers cannot read it.

Step 2: curl the page and grep for your facts

The faster version, and the one you can repeat across a site. It fetches the page with the OAI-SearchBot user-agent, because some sites serve bots different HTML, then counts what is there:

curl -s -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot" https://www.example.com/pricing -o pricing.html

grep -c -F '$49' pricing.html
grep -o -i -E "starter|growth|enterprise" pricing.html | sort | uniq -c
grep -o -i -E "noindex|nosnippet|max-snippet|data-nosnippet" pricing.html

Swap in your own price and plan names. A zero for a price you can see on screen means that price is rendered in the browser and nowhere else. The last line looks ahead to the next section.

Step 3: check content behind tabs, accordions and "load more"

Interactive content splits into two cases. If the text is in the raw HTML and only collapsed by CSS, the crawler at least receives it. If it is fetched from an API when someone clicks a tab, expands an accordion or hits "load more," it is not in the HTML, and a non-rendering crawler never sees it. Same mechanism as client-side rendering, different fix. Test it the same way: copy a sentence from inside a closed tab and grep for it in the raw file.

Fixing what you find

In order of preference: render the fact-bearing pages on the server or generate them statically; prerender those routes if a full change is not on the table this quarter; and at minimum, put the load-bearing facts, meaning prices, plan names and what the product does, into the initial HTML even if the rest of the page stays client-side. Then run the grep again. A fix you did not re-test is a guess.

Directives that stop engines indexing or quoting

The second silent failure is a directive. Four of them matter, and none has an additive counterpart. There is no tag that makes a page more quotable. These only take quotability away.

Directive	Where it lives	What it does
noindex	Robots meta tag or X-Robots-Tag header	Keeps the page out of the index of engines that honor it, Google and Bing included. A page outside Google's index cannot appear in AI Overviews or AI Mode.
nosnippet	Robots meta tag or X-Robots-Tag header	No text from the page may be shown as a snippet. Google requires snippet eligibility for AI Overviews and AI Mode, so this removes the page from both.
max-snippet:[n]	Robots meta tag or X-Robots-Tag header	Caps how many characters may be quoted. max-snippet:0 is the same as nosnippet.
data-nosnippet	An attribute on one HTML element	Keeps that element's text out of snippets and AI summaries while the page stays indexed.

In the page they look like <meta name="robots" content="noindex">, <meta name="robots" content="max-snippet:0"> or <div data-nosnippet>. In a response header, X-Robots-Tag: nosnippet.

How these get applied by accident

Nobody writes noindex on their pricing page on purpose. It arrives some other way.

• From a template. A shared layout carries a robots meta tag added for one page type, and every page on that layout inherits it.
• From staging. The noindex that kept a staging site out of Google ships to production with a deploy.
• From a cookie banner. Someone wraps a container in data-nosnippet to keep banner text out of snippets, and the container also holds the main content.
• From the server or CDN. An X-Robots-Tag header set at the edge never appears in the page source, so a source review misses it. Run curl -s -I https://www.example.com/pricing and read the headers.
• From a plugin. An SEO plugin setting switched on for a whole post type covers every post of that type.

The suppression scan in the 99Visibility Crawler Access check reads both the HTML and the response headers for noindex, nosnippet, max-snippet:0 and data-nosnippet. For data-nosnippet it shows a sample of the hidden text, so you can tell a cookie banner from a pricing table.

The Search Console control you cannot see from outside

Google added a Search Console control for its generative AI features. It is opt-out, not opt-in: inclusion "is the default control for all properties." It covers AI Overviews, AI Mode and generative AI features in Google Discover. For a child property the default is "Inherit control from parent," so a property can be excluded by a setting on a parent property that someone else manages.

No crawl can detect this, ours included. There is no tag, no header and no API. If you have Search Console access, find the setting and confirm inclusion is on for the property and for its parent. Google is still rolling it out to a subset of owners, so you may not see it yet. If an agency or a former employee set up the parent property, ask them. Google states the control is not used as a ranking or inclusion signal elsewhere in Search, and it is separate from Google-Extended, which governs Gemini.

The 2MB ceiling: real, rare, report it only when hit

Google documents a hard limit on how much of a page Googlebot reads. Gary Illyes on Google's Search Central blog (March 31, 2026): "Googlebot currently fetches up to 2MB for any individual URL (excluding PDFs). … Any bytes that exist after that 2MB threshold are entirely ignored." The same post gives PDFs 64MB and says any other crawler that does not specify a limit defaults to 15MB. The counter is per URL; images, scripts and stylesheets are counted separately.

Google's own framing matters as much as the number: "For the vast majority of the web, a 2MB HTML payload is massive, and you will never hit this limit." This is a Googlebot fact, and it says nothing about how much ChatGPT's or Perplexity's crawlers read. Check the size of the HTML file your curl saved. If it is under 2MB, drop the topic. If a page inlines a huge data payload and goes over, that page has a real problem. Anything else is selling a limit nobody hit.

Common mistakes

• Checking in a browser. Your browser runs JavaScript. The AI crawlers do not. Use view-source or curl.
• Using Google as the test. Googlebot renders. A page that ranks well can be empty to ChatGPT, Claude and Perplexity.
• Reading only the page source for directives. X-Robots-Tag lives in the response headers. Check both.
• Wrapping a layout container in data-nosnippet. It hides everything inside, including what you want quoted.
• Shipping staging's noindex to production. Make the robots tag depend on the environment, and check the live page after every deploy.
• Assuming the Search Console control is on. It is on by default, but child properties inherit from the parent. Look.
• Adding markup to help AI read the page. No tag makes content more quotable. These controls only subtract.
• Reporting the 2MB limit for pages nowhere near it. Report it only when a page exceeds it.