We Audited Our Own Site. Google Indexes 39% of It.
Google indexes 52 of the 132 URLs in our sitemap. That is 39%, on a site that sells AI visibility audits, so we ran the audit on ourselves.
Nothing was blocked. The problem was what we had published, plus one framework default we had never looked at.
Step one: rule out the technical failures
We fetched every URL in the sitemap. All 132 returned 200, none carried noindex, and every canonical pointed at the page itself.
When every URL is technically clean and Google still skips most of them, the problem is not access. It is Google's judgement of the pages.
Page count is not the problem either. Google's crawl budget guide is written for sites with "1 million+ unique pages", or 10,000+ pages that change daily (Google, updated 2026-07-22). At 132 pages, Google can crawl everything we have.
More indexed pages did not bring more impressions
On 17 September we fixed how our pages link to each other. Ten blog posts had no inbound link from any other post, and seven docs pages were reachable only through a chain of next and previous links.
Google picked the pages up fast. Indexed pages went from 34 to 48 in two days and 53 in five, then held at 52 to 53 for three weeks.
Impressions did not follow. We averaged 139 a day in the 28 days before the fix and 132 a day after it.
| Period | Pages indexed | Impressions per day |
|---|---|---|
| 21 Aug to 17 Sep (before the fix) | 30 to 34 | 138.5 |
| 22 Sep to 12 Oct (after) | 52 to 53 | 132.2 |
Getting a page indexed and getting it shown are different problems. Fixing discovery solved the first. It did nothing for the second: the 19 new pages either answer questions few people search for, or rank too low to be seen.
What the skipped pages have in common
We measured each page's own text: the words in its main content, minus navigation, and minus any paragraph that appears on four or more pages.
| Words of its own | Pages |
|---|---|
| Under 150 | 12 |
| 150 to 299 | 27 |
| 300 to 599 | 33 |
| 600 to 999 | 35 |
| 1,000 or more | 25 |
39 pages have fewer than 300 words of their own. The thinnest sections were the ones closest to the product: feature pages (median 149 words), free tools (190) and help articles (214).
The same topic, written six times
Short pages were the smaller problem. Grouping pages by title and slug, we had written the same 9 topics three to six times each, spread across the blog, academy, docs, help and resources.
| Topic | Pages covering it |
|---|---|
| What GEO is, and GEO vs SEO | 6 |
| Best GEO tools | 5 |
| AI hallucinations about your brand | 5 |
| Tracking competitors | 5 |
| Understanding your score | 4 |
| Schema markup | 4 |
| How ChatGPT talks about brands | 4 |
| Measuring ROI | 3 |
| Checklists | 3 |
When two of your pages answer the same question for the same reader, you are asking Google to choose between them. A feature page and a docs page on one topic can both earn their place, because one sells and one explains. Two explainers of the same thing cannot.
The finding we didn't expect: our titles were in the body
The first script we wrote for this audit reported the title of every page as "99Visibility". The first <title> tag in our HTML was the one inside our SVG logo. The real page title came more than 28,000 characters after <head> had closed.
Since Next.js 15.2, the framework streams page metadata (title, description, canonical, Open Graph tags) into the <body> so pages start rendering sooner. User agents on its htmlLimitedBots list get the metadata in <head> instead (Next.js docs). We requested one of our pages as each crawler and recorded where the tags landed.
It only happens on pages rendered on demand, which was 84 of our 132 URLs. Pages built ahead of time already had their metadata in the head.
| Crawler | Before the fix | After |
|---|---|---|
| Googlebot | Body | Head |
| Bingbot | Head | Head |
| GPTBot | Body | Head |
| OAI-SearchBot | Body | Head |
| ClaudeBot | Body | Head |
| PerplexityBot | Body | Head |
Googlebot runs JavaScript, so after rendering it sees the tags moved back into <head>. The AI crawlers do not. Vercel's study of AI crawler traffic found that ChatGPT's and Claude's crawlers fetch JavaScript files but "don't execute them" (Vercel).
What GPTBot or ClaudeBot does with a title it finds in the body is undocumented. A spec-compliant HTML parser still finds the right one; our first script didn't. No AI company publishes which kind of parser its crawler runs, so nobody outside those companies can tell you the effect.
Google is explicit about one tag. The canonical link "is only accepted if it appears in the <head> section of the HTML", and Google asks you to set it in the HTML source rather than have JavaScript change it (Google). Our raw HTML had it in the body and relied on JavaScript to move it.
The one-line fix, and the trap next to it
// next.config.ts
htmlLimitedBots: /.*/,This serves metadata in <head> to every request. Next.js documents it as the way to turn streaming metadata off.
Don't list only the AI bots. Setting htmlLimitedBots replaces Next's default list instead of adding to it. A regex of GPTBot|ClaudeBot|PerplexityBot would move Bingbot, Twitterbot, LinkedInBot and Slackbot back to streamed metadata. The docs say so, and we confirmed it in the framework's source.
The trade-off: a page now waits for its metadata before it starts streaming. If your metadata comes from a slow API call, measure your page speed before and after.
Check your own site in ten minutes
- Compare two numbers. Count the URLs in your sitemap, then open the Pages report in Google Search Console and read the Indexed count. Ours were 132 and 52.
- Rule out access first. Run five of the non-indexed URLs through Search Console's URL Inspection. It shows the status, any
noindexand the canonical Google chose. - See where your title lands. In Chrome, open
view-source:followed by your page URL. Search for</head>, then search for your page title. If the title comes after</head>, your metadata is being streamed. - Count what is yours on each page. Ignore the navigation and the footer. A page you want to rank with fewer than 300 words of its own is where to start.
- Write down the question each page answers. If two pages answer the same one for the same reader, give each a different job.
What we're changing
- Every page gets one job. Help articles answer "I'm stuck, what now?" Docs are the feature-by-feature reference. The blog and academy teach.
- Feature pages and free tools get real product output, not filler. They are the pages that make money, and they were the thinnest on the site.
- Metadata sits in
<head>for every crawler, using the line above.
Frequently Asked Questions
Does having too many pages hurt SEO?
Not by count. Google's crawl budget guidance starts at a million pages. What matters is how many of your pages are worth indexing, and whether they compete with each other for the same question.
Does Next.js streaming metadata hurt Google rankings?
We have no evidence it does. Googlebot renders JavaScript, and Next.js designed streaming around that. We turned it off because of Google's rule on canonicals and because the AI crawlers don't render.
Will moving metadata into the head get us cited by ChatGPT?
No one has measured that. Metadata in <head> removes one way to be misread. It does not make a page more likely to be chosen.
Check the access side automatically
The checks above are manual. A 99Visibility audit runs the access side for you: it requests your site as each AI crawler, reads your robots.txt rules bot by bot, and flags noindex and nosnippet. These checks decide whether a crawler gets your page at all. The free plan includes one audit a month. Run your free audit.
Want more of our research in Google Search and AI Mode? Add us as a preferred source.
Can AI crawlers reach your site?
Your free audit requests your site as each AI crawler, reads robots.txt bot by bot, and shows every place your firewall disagrees with it.
Start Free Audit2 of 3 retrieval crawlers can reach you
robots.txt allows 1 crawler your firewall blocks
Related Articles
Should I Block AI Crawlers on Cloudflare? A Per-Bot Decision Guide
Block training crawlers if you object to training; keep search and agent crawlers open to stay in AI answers. What Cloudflare changed on September 15, 2026, per crawler, and how to check yours.
LLM Brand Visibility: How AI Models Decide What to Say About You
An AI answer about your brand passes four stages: memory, crawler access, retrieval and generation. What each AI vendor does at each one, and the stage you control.
What Is llms.txt? The 2026 Spec, a Real Example, and Who Reads It
llms.txt is a Markdown summary for AI agents. Here is the v2 spec, a correct example file, the log data on who fetches it, and why Google says Search ignores it.