Create SEO-friendly robots.txt files to control search engine crawlers. Set crawl rules, sitemaps, and crawler delays for better website indexing.
Robots Configuration
Quick Presets
Basic Settings
Search Engine Bots
Select specific bots below or use "All Bots" for universal rules
Google Search Engines
Microsoft Search Engines
Yahoo Search Engines
Other Search Engines
Specialized Crawlers
Allow/Disallow Rules
Common Directories to Block
Additional Options
Generated Robots.txt
Ready to Generate
Configure your crawler settings and click "Generate Robots.txt" to create your file
Robots.txt Best Practices
- Place robots.txt in your website's root directory
- Use specific paths rather than wildcards when possible
- Include your sitemap URL for better crawling
- Test your robots.txt with Google Search Console
- Be careful not to block important pages accidentally
- Update robots.txt when your site structure changes