Applebot vs Applebot-Extended: What Blocking Each One Costs You

Applebot is Apple's web crawler. Its data powers search in Spotlight, Siri and Safari, may be used to train Apple's foundation models, and may supply context for AI-generated answers in Siri and Search. Applebot-Extended is not a crawler at all: it is a robots.txt token that opts your content out of training while leaving you in Apple's search results.

So blocking Applebot and blocking Applebot-Extended are different decisions with different costs. Everything below comes from Apple's About Applebot page (published September 4, 2026) and Apple's published IP list, as of October 2026. Apple is the third vendor in this series, after OpenAI's three crawlers and Anthropic's ClaudeBot and its siblings, and it splits the job differently from both.

What Applebot does, per Apple

Apple describes three uses for the same crawl:

• Search. "The data crawled by Applebot is used to power various features, such as the search technology integrated into many user experiences in Apple's ecosystem including Spotlight, Siri, and Safari. Enabling Applebot in robots.txt allows website content to appear in search results for Apple users around the world in these products."
• Training. The data "may also be used to help train Apple foundation models powering generative AI features across Apple products, including Apple Intelligence, Services, and Developer Tools."
• AI answers. Applebot data "may be used to provide additional context and up-to-date content when AI models are used to generate output", for example "answering broad world knowledge questions in Siri and Search that may include links to sources and websites used to help generate the answer."

OpenAI and Anthropic each run a separate crawler per job. Apple runs one crawler and gives you three separate switches to limit what it does with the result.

Three controls, three different costs

What you want to stop	Apple's control	What you keep, per Apple
Training Apple's foundation models	Disallow for Applebot-Extended in robots.txt	Search. "Webpages that disallow Applebot-Extended can still be included in search results." Its rules "are not considered in ranking for Search."
AI answers in Siri and Search using your page	nosnippet robots meta tag or X-Robots-Tag header	Listing. Suggestions to visit the URL "will only include the page's title."
AI answers using a paywalled page	isAccessibleForFree: false in page-level JSON-LD	Search. Such pages "are eligible to appear in search results", but Applebot "will not use that content as additional context" for AI output
Appearing in Spotlight and Siri Suggestions	noindex	Nothing on that page: "it won't appear in Spotlight or Siri Suggestions"
Everything	Disallow for Applebot in robots.txt	Nothing. Apple ties appearing in its search results to allowing Applebot

Read the first row carefully. Disallowing Applebot-Extended does not keep your content out of AI answers in Siri. Apple says so directly: even if you disallow Applebot-Extended and add nosnippet, Applebot may still crawl and "your content will remain discoverable through Spotlight, Siri, and Safari." Training, AI answers and search are three settings, not one.

Applebot-Extended crawls nothing

"Applebot-Extended does not crawl webpages. … Applebot-Extended is only used to determine how to use the data crawled by the Applebot user agent." You will never see it in your logs, and there is no IP to allow or block. Google-Extended works the same way; our AI crawler list covers both tokens.

The nosnippet trap

Apple's AI-answer opt-out is a standard tag, which means a generic version of it reaches other search engines too. <meta name="robots" content="nosnippet"> also tells Google to keep the page out of its snippets, and Google says nosnippet "will also prevent the content from being used as a direct input for AI Overviews and AI Mode" (Google robots meta tag docs). To scope the opt-out to Apple, Apple's own examples name its crawler: <meta name="applebot" content="nosnippet">, or the header X-Robots-Tag: applebot: nosnippet.

The Googlebot fallback

This is the line in Apple's page most robots.txt files trip over: "If robots instructions don't mention Applebot but mention Googlebot, the Apple robot will follow Googlebot instructions."

If your file has a User-agent: Googlebot group and no Applebot group, every Disallow you wrote for Google also applies to Apple. A staging path blocked for Googlebot is blocked for Applebot. A site that blocks Googlebot from / to stay out of Google is also out of Spotlight, Siri and Safari search. Two other details from the same page: Applebot "does not follow crawl-delay", and it "may render the content of your website within a browser", so JavaScript and CSS blocked in robots.txt can stop it rendering the page properly. That differs from ChatGPT's and Claude's crawlers, which Vercel found fetch JavaScript files but don't execute them.

robots.txt examples

Out of Apple training, in Apple search

User-agent: Applebot
Allow: /

User-agent: Applebot-Extended
Disallow: /

Out of Apple entirely

User-agent: Applebot
Disallow: /

Give Applebot its own group whenever your file has a Googlebot group, so the fallback never decides for you. Our WordPress robots.txt guide covers where the file lives if a plugin writes it.

The firewall gate

robots.txt is a request. Your CDN or firewall decides what Applebot actually receives. Apple's page says nothing about firewalls, so this part is our inference: a 403 or a challenge page served to Applebot keeps the page out of Apple's index whatever your robots.txt allows, the same way it does for OpenAI's and Perplexity's crawlers.

One CDN default makes this likely. Since September 15, 2026, Cloudflare's Training setting of "Block" also blocks Applebot, Googlebot and Bingbot, because Cloudflare classes them as mixed-use; our Cloudflare post has the dashboard steps. The setting that refuses training and keeps search is Disallow AI Training.

Verify real Applebot traffic in two ways Apple documents: reverse DNS, where genuine requests resolve to *.applebot.apple.com, or the Applebot IP list. That list held 24 IPv4 prefixes and no IPv6 prefixes when we read it in October 2026, with a creation time of September 15, 2026. Re-fetch it on a schedule rather than hard-coding it.

One Apple agent ignores robots.txt by design: "The iTMS user agent does not follow robots.txt, as it is not a general search crawler". It only crawls URLs registered with Apple Podcasts.

Check your site in six steps

1. Open your live robots.txt and find the Applebot, Applebot-Extended, Googlebot and * groups.
2. If there is no Applebot group, read the Googlebot group as Apple's. That is what Applebot will obey.
3. Look for Disallow lines on JavaScript, CSS or API paths that a page needs to render.
4. Search your page source and headers for noindex, nosnippet and none, in a robots or applebot meta tag or an X-Robots-Tag header: curl -sI https://www.example.com/ | grep -i x-robots-tag
5. Request a page as Applebot and check for a 200 with your real page in it: curl -sL -o page.html -w "%{http_code}" -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)" https://www.example.com/. A test from your machine cannot exercise IP-based rules, so a 200 here is indicative, not proof.
6. Check your logs and CDN for requests from Apple's IP ranges and the status codes you returned, and confirm your CDN's AI crawler setting is not Block.

The free robots.txt checker reads your file for Applebot-Extended alongside the other AI crawlers and shows the line that decides each one. For Applebot itself, steps 2 and 5 are the check.

What is unmeasured

Apple publishes no figures for how many answers in Siri or Spotlight draw on web pages, how much traffic they send, or how often Applebot crawls a typical site. Apple does list the factors Apple Search may use to rank results, including engagement, relevance and links, with "no (pre-determined) importance of ranking", so there is no weight to tune. Allowing Applebot buys eligibility for Apple's surfaces. Nothing in Apple's documentation says it buys placement.

FAQ

What is Applebot?

Applebot is Apple's web crawler. Apple says its data powers search in Spotlight, Siri and Safari, may help train Apple's foundation models, and may supply context for AI-generated answers in Siri and Search.

Should I block Applebot?

Only if you want out of Apple's search features. Apple ties appearing in Spotlight, Siri and Safari results to allowing Applebot. To refuse training only, disallow Applebot-Extended and keep Applebot allowed.

Does blocking Applebot-Extended remove me from Siri?

No. Apple says pages that disallow Applebot-Extended can still be included in search results, and its rules are not considered in ranking. Keeping your content out of AI answers in Siri and Search takes nosnippet, not robots.txt.

Does Applebot respect robots.txt?

Yes, for general search crawls, and it falls back to your Googlebot rules when you do not name it. It does not follow crawl-delay. The separate iTMS agent for Apple Podcasts does not follow robots.txt.

For the other vendors, see Perplexity's two crawlers, GPTBot vs OAI-SearchBot and ClaudeBot.

See whether AI crawlers can reach you. Run a free audit: it compares what your robots.txt says with what your firewall returns to each AI crawler. No credit card.