Based on what buyers actually ask
Auto-generated tagsFeatureAI-powered
robots.txt for AI
robots.txt for AI, audited bot by bot and page by page
Your robots.txt for AI decides which AI bots may fetch which pages, and a rule written for one purpose often blocks another. WriteWorks reads the file the way each AI bot does, applies it to every indexable page, and shows the share of your site each bot can reach. You can block AI crawlers that collect training data and keep the bots that put you in AI search answers, with the audit confirming which is which.
7-day trial · Cancel anytime
Google Search Console
Google Analytics 4
Google Ads
Ahrefs
Semrush
DataForSEO
Firecrawl
Tavily
Exa
Parallel
G2
WordPress
Webflow
Contentful
Sanity
Payload
BaseHub
Framer
Notion
Gamma
v0
Slack
Linear
Atlassian
Monday.com
Granola
Klaviyo
Customer.io
OpenAI Ads
Claude and MCP clients The platform
See WriteWorks in Action
This is the working product, not a mockup: the same visibility scores, briefs and published content your team would be looking at. Start a free trial and point it at your own domain, or take the tour of what sits behind the screen.
Home
Ask AI
Opportunities128
Research
Create
Measure
AI visibility
AI discoverabilityNew
AI crawlers
Reports
Context
Support
Settings
Sign out
SCSam Cartersam@example.com
WRITEWORKS
Measure/AI discoverability
Search brands, prompts, competitors…⌘K
New1.2M (62%) leftAsk AI
AI discoverability
Measure
Overview
Issues
Pages
Bots
Settings
The challenge
Why AI bot access goes wrong
Rules written for one bot, read differently by another.
One rule, unintended consequences
A Disallow added to stop model training can also stop the bot that fetches pages for live answers, because the two belong to the same company and the rules are easy to conflate.
robots.txt is read differently by each bot
A bot follows the group that names it and ignores the * group. A file that looks permissive under * can still shut out a named AI bot.
Blocked at the server, not the file
A firewall or bot-protection rule can refuse AI user agents even when robots.txt allows them, and nothing in the file shows it.
Pages are built for links, not answers
Content written to rank in blue links rarely carries the structure an assistant needs to quote it directly.
The solution
How the AI bot access audit works
Evaluate the file per bot, apply it per page, and catch server refusals too.
WriteWorks evaluates robots.txt for each tracked bot against every indexable page, following the RFC 9309 rules: a bot obeys the group that names it, falling back to * only when none does; the longest matching rule wins; Allow wins a tie; and * and $ wildcards are honoured.
Pages blocked from all AI search bots score zero for bot access, and pages blocked from some lose points by the share blocked. Blocking training bots never costs score; an inconsistent training policy is reported so you can make it deliberate.
Each page is also requested with an AI user agent. If that request is refused with a 401, 403, 429 or 503 while a normal fetch succeeds, the page is flagged as one your server may refuse to AI.
Open any bot to see the pages it can fetch, the pages it can't (most important first) and the exact robots.txt rule blocking each one, plus the group and rules it obeys.
Success story
What real brands fixed with WriteWorks
What each brand fixed, measured before and after with the same audit, in their own words.
“Nobody had spotted that our market homepages carried no structured data at all. Now 100% of them tell AI exactly who GraftPal is, and every FAQ page in the audit carries a modified date.”
GP
Capabilities
What you can see
Access policy, by bot and by page.
Bots view
Every tracked bot by role, with pages allowed, pages blocked and the change since the last crawl.
Bot access over time
Trend of each bot's access across crawls.
Bot detail
Blocked pages with the blocking rule, and the group the bot obeys.
Real visits alongside
With crawler logs connected, 90-day visits and last seen for each bot.
Choose the bots you track
Pick which search and training bots count in settings.
Per-page verdicts
A robots.txt verdict for every bot on every page detail view.
Time savings
What changes
robots.txt decisions you can check.
TaskBeforeWith WriteWorksTime saved
Check robots.txt for AI botsRead the file, simulate each botPer-bot access across the siteThe simulation
Block AI training crawlers safelyEdit and hopeSee search access unchanged after the next crawlLost visibility
Find server-level blocksTest user agents by handRefusals flagged per pageHours
Included
What's included
The bot access pillar of AI discoverability.
18 bots in four roles: AI search indexes, live answer fetchers, search engines feeding AI, model training
robots.txt evaluated per bot following RFC 9309
Pages blocked from all or some AI search bots, scored
Training bot policy reported, never scored
Server refusals of AI user agents detected
Blocked pages listed with the rule that blocks them
Bot access trend across crawls
90-day visits per bot when crawler logs are connected
Built for
Who uses it
Everyone with a say in who reads the site.
Technical SEO
robots.txt decisions checked against every page, for every AI bot.
Legal and policy owners
A clear record of which bots may train on your content and which may not.
Developers and ops
Spot firewall rules that refuse AI user agents.
Featured story
GoTeachingJobs
“Honestly, the jobs board looked fine in a browser, so we never questioned it. The audit showed that only 5% of our job pages gave crawlers real content. We moved the listings into the server HTML, and the re-audit came back at 100%. The typical job page now carries 73% of its content before any JavaScript runs, and we're working through the rest.”
GoTeachingJobs team
Read the full story5% → 100%
Job pages AI crawlers can read
18% → 100%
Median share of page content in the HTML crawlers get
45% → 99%
Pages passing the answer-first check
FAQ
Frequently asked questions
Everything teams typically ask before getting started.
How do I write robots.txt for AI bots?+
Give each AI bot you want to control its own User-agent group, because a bot that finds a group naming it ignores the * group entirely. Decide separately for search and answer bots, which fetch pages to cite them, and training bots, which collect content for models. Then check the result: WriteWorks evaluates the file for every tracked bot across every indexable page and shows what each one can actually reach.
Can I block AI crawlers without disappearing from AI search?+
Usually, yes, because most AI companies use separate bots for training and for search or live answers. You can block AI crawlers that train models, such as GPTBot, ClaudeBot or Google-Extended, while allowing OAI-SearchBot, Claude-SearchBot, PerplexityBot and the user-triggered fetchers. WriteWorks scores only search access, so a training block costs nothing, and the next crawl confirms search access is intact.
Which bots does the audit check?+
AI search indexes: OAI-SearchBot, Claude-SearchBot and PerplexityBot. Live answer fetchers: ChatGPT-User, Claude-User, Perplexity-User and Google-Agent. Search engines feeding AI: Googlebot, Bingbot, Applebot and Amazonbot. Model training: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent and Bytespider.
What happens if robots.txt can't be fetched?+
The crawl follows the standard: a 4xx response means everything is allowed, while a 5xx response or an unreachable file means everything is treated as blocked, which is worth fixing fast.
How is this different from AI crawler tracking?+
This audit reports what your rules allow. AI crawler tracking reports what bots actually do, from your own traffic. Connect both and the Bots view shows each bot's access next to its real visits over 90 days.
Does blocking a bot in robots.txt stop it completely?+
robots.txt is a request, honoured by well-behaved crawlers. It doesn't enforce anything. To see whether a bot still visits after you block it, use AI crawler tracking, which counts real requests.
Four roles
Not every AI bot does the same job
The audit groups bots by what they do, because the right robots.txt policy depends on it.
AI search indexes
OAI-SearchBot, Claude-SearchBot, PerplexityBot. Blocking these keeps pages out of those AI search indexes, and costs score.
Live answer fetchers
ChatGPT-User, Claude-User, Perplexity-User, Google-Agent. They fetch a page when someone asks.
Search engines feeding AI
Googlebot, Bingbot, Applebot, Amazonbot. Their indexes sit under several AI answer experiences.
Model training
GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider. Block them at no score cost.
Explore more
Keep exploring
Related solutions, adjacent use cases, and platform features.
Check your robots.txt for AI before it costs you
Every AI bot, every page, the rule behind every block, scored inside the AI SEO audit.