Free tool · no signup
Choose which AI crawlers can read your site
Allow or block each AI crawler and copy the robots.txt lines. We warn you when a choice keeps an AI engine from reading your site.
Which AI crawlers may read your site?
Search and user crawlers put you in answers. Training crawlers only feed models, so blocking them costs you nothing in answers.
Your robots.txt lines
Add them to robots.txt at your site's root. Rules for other bots can stay as they are.
# AI crawlers (made with CiteBrief's free generator) User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: GPTBot Allow: / User-agent: Googlebot Allow: / User-agent: Google-Extended Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Bingbot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: ClaudeBot Allow: /
Search crawlers and training crawlers
AI engines send two kinds of crawler. Search and user crawlers (OAI-SearchBot, PerplexityBot, Googlebot) fetch your pages to answer a question right now; block them and you drop out of those answers. Training crawlers (GPTBot, Google-Extended, ClaudeBot) collect text to train models; blocking them doesn’t change what engines answer today.
robots.txt is a request, not a lock, and a firewall or CDN can still turn crawlers away. Check what your server actually does.