Robots.txt Generator
Build a robots.txt file visually — configure crawl rules for search engines, block specific directories, and add your sitemap URL. Download ready to upload.
What Is robots.txt?
The robots.txt file is a plain text file placed at the root of your website (e.g. https://example.com/robots.txt) that tells search engine crawlers which pages or directories they are allowed or not allowed to crawl. It follows the Robots Exclusion Protocol. While robots.txt is a courtesy — crawlers are not required to obey it — all major search engines (Google, Bing, DuckDuckGo) respect it by default.
Key robots.txt Directives
User-agent specifies which bot the rule applies to. * applies to all bots. Googlebot applies only to Google's crawler. Disallow blocks a path from being crawled. Allow explicitly permits a path (useful when a parent is disallowed). Sitemap tells crawlers where to find your XML sitemap — a significant SEO signal. Crawl-delay asks bots to wait N seconds between requests (not honoured by Google).
Important Limitations
Blocking a URL with robots.txt does not prevent it from appearing in search results if other pages link to it — Google may still index the URL but cannot read the content. To prevent indexing, use a noindex meta tag on the page itself. Also, sensitive data should never be protected only by robots.txt — it is not a security mechanism, just a crawl instruction.