robots.txt Generator
Generate robots.txt files for SEO and crawler control. Configure which paths search engine bots can access with template support.
Add Rule
Add your sitemap URL to help search engines discover your content
Delay between crawler requests (not supported by all bots, Google ignores this)
Preview
# Configure rules above to see preview
Installation Instructions
Generate your robots.txt file
Configure rules above and download the generated robots.txt file.
Upload to your website root
Place the robots.txt file in the root directory of your website (e.g., https://example.com/robots.txt).
Test your robots.txt
Visit https://yoursite.com/robots.txt to verify it's accessible. Use Google Search Console's robots.txt Tester to validate.
Monitor and update
Regularly review and update your robots.txt as your site structure changes.
Template Support
Pre-built templates for common scenarios like WordPress and e-commerce sites.
Syntax Validation
Real-time validation ensures your robots.txt follows RFC standards.
Live Preview
See exactly how your robots.txt will look before downloading.
Frequently Asked Questions
What is a robots.txt file?
A robots.txt file is a text file placed in the root of a website that tells search engine crawlers which pages or sections of your site they can or cannot access. It follows the Robots Exclusion Protocol and helps manage crawler traffic and SEO.
What is the difference between Allow and Disallow?
Disallow tells crawlers not to access specified paths, while Allow explicitly permits access. Allow is typically used to override a broader Disallow rule for specific files or directories. For example, you might disallow /admin/ but allow /admin/public/.
What does User-agent: * mean?
User-agent: * is a wildcard that applies rules to all web crawlers and bots. You can also specify individual bots like Googlebot, Bingbot, etc., to apply different rules to different crawlers. This is useful when you want to give different access permissions to different search engines.
Should I block /wp-admin/ in WordPress?
Yes, it's recommended to block /wp-admin/ for security and SEO reasons. However, you should allow /wp-admin/admin-ajax.php as it's needed for AJAX functionality on your site. The WordPress template includes these best practices.
What is Crawl-delay?
Crawl-delay specifies the number of seconds a crawler should wait between requests to your server. It helps prevent server overload from aggressive crawlers. Note that this directive is not supported by all crawlers - notably, Google ignores it completely.
Where should I place my robots.txt file?
The robots.txt file must be placed in the root directory of your website (e.g., https://example.com/robots.txt). It won't work in subdirectories, with a different filename, or in a different location. Crawlers always look for it at the root.
Can robots.txt block all access to my site?
robots.txt tells well-behaved crawlers what not to access, but it's not a security measure. Malicious bots can ignore it completely. For truly private content, use proper authentication methods like password protection, HTTP authentication, or server-side access controls.
Should I include my sitemap in robots.txt?
Yes, adding your sitemap URL to robots.txt helps search engines discover and index your content more efficiently. Use the Sitemap directive followed by the full URL to your sitemap.xml file. You can include multiple sitemap URLs if needed.