What is a Robots.txt File?

The robots.txt file is a simple text file placed in the root directory of your website. It acts as a set of instructions for web robots (also known as crawlers or spiders), telling them which pages they are allowed to crawl and index, and which ones they should ignore.

Why Block AI Bots?

With the rise of Generative AI, many website owners are concerned about their content being used to train Large Language Models (LLMs) without permission. Bots like GPTBot (OpenAI), Google-Extended (Gemini), and CCBot (Common Crawl) scour the web for data. Using this generator, you can easily add rules to block these specific crawlers while still allowing search engine bots like Googlebot to index your site for SEO.

Key Directives Explained

• User-agent: Specifies which bot the rule applies to (e.g., User-agent: * applies to all bots).
• Disallow: Tells the bot not to visit a specific path or folder (e.g., Disallow: /admin/).
• Allow: Overrides a disallow rule for a specific sub-path (e.g., Disallow /admin/ but Allow: /admin/public-image.jpg).
• Sitemap: Points crawlers to your XML sitemap to help them find content faster.

How to Use This File

Once you have generated your code:

1. Download the file as robots.txt.
2. Upload it to the main folder (root directory) of your website via FTP or your hosting file manager.
3. The final URL should look like https://yourwebsite.com/robots.txt.
4. Test it using the Google Search Console Robots Testing Tool to ensure you aren't accidentally blocking important pages.