Create a robots.txt file for your website in seconds. Control how search engines and AI crawlers access your content.
Quick Presets
robots.txt
User-agent: * Allow: /
https://yoursite.com/robots.txtAI companies like OpenAI (GPTBot), Google (Google-Extended), and others crawl websites to train their models. You can block these crawlers using the "Block AI Crawlers" preset. Note that compliance with robots.txt is voluntary — not all crawlers respect it.
Preconfigured rules for Googlebot, Bingbot, GPTBot, and 10+ other crawlers.
One-click preset to block AI training crawlers like GPTBot, Google-Extended, and CCBot.
Start with Allow All, Block All, Block AI, or WordPress presets and customize from there.
Download your robots.txt file directly or copy the content to your clipboard.
No signup. No credit card. No limits. Generate as many files as you need.
Learn what each directive does with built-in explanations and best practices.
A robots.txt file tells search engine crawlers which pages or files on your site they can or can't request. It's a standard used by websites to communicate with web crawlers and bots.
The robots.txt file must be placed in the root directory of your website. For example: https://example.com/robots.txt. It won't work if placed in a subdirectory.
Robots.txt can prevent crawlers from accessing your pages, but Google may still index the URL (without content) if other sites link to it. For guaranteed removal from search results, use the 'noindex' meta tag instead.
That depends on your preference. If you don't want AI companies using your content to train their models, you can block crawlers like GPTBot (OpenAI), Google-Extended, and CCBot. Use our 'Block AI Crawlers' preset for a quick setup.
Crawl-delay tells bots to wait a specified number of seconds between requests. This can prevent your server from being overwhelmed. Note: Googlebot ignores Crawl-delay — use Google Search Console to set crawl rate for Google.
Robots.txt is a voluntary standard. Most legitimate search engine bots follow it, but malicious bots may ignore it. It should not be used as a security measure — use proper authentication and access controls for sensitive content.
Generate your robots.txt in under a minute. Free forever.
Generate Robots.txt Now