Robots.txt Generator
Build a robots.txt file with common presets, custom rules, sitemap inclusion, and AI crawler controls.
Try Robots.txt Generator →What It Does
The Robots.txt Generator builds a properly formatted robots.txt file for your website. It includes presets for common platforms (WordPress, Shopify), AI crawler control (block or allow GPTBot, Claude-Web, PerplexityBot), and a custom rule builder for any user-agent.How to Use It
- Navigate to Generators → Robots.txt Generator.
- Click a preset to start with common configurations.
- Add your sitemap URL (recommended).
- Add or edit User-Agent rules with disallow and allow paths.
- Click Generate robots.txt.
- Copy or download the file and upload to your site root.
AI Crawler Control
2seo includes presets for both blocking and explicitly allowing AI training crawlers. Use these to control whether GPTBot, Claude-Web, PerplexityBot, and Google-Extended can access your content. Allowing them increases your chances of being cited in AI Overviews and chat responses.
Use Cases
- New site setup: Generate a baseline robots.txt before launch.
- AI control: Block or welcome AI training crawlers.
- WordPress/Shopify: Use platform-specific presets to avoid common mistakes.
- Privacy: Block specific bots from indexing private pages.