robots.txt Generator

Technical details

robots.txt is a plain-text file served at the root of your domain (https://example.com/robots.txt) that tells search engine crawlers which URLs they may request. It is a contract of trust — well-behaved bots respect it, malicious bots ignore it. robots.txt does not prevent indexing of URLs already known to Google; for that you need a `noindex` meta tag or HTTP header.

A robots.txt file is structured into groups, each starting with one or more `User-agent` lines followed by `Allow` and `Disallow` directives. Google honours the longest matching path, so `/private/` is more specific than `/`. The generator builds the directives in the right order, prevents common mistakes (`Disallow: /` blocks the whole site), and adds a `Sitemap:` line at the end.

Comments start with `#` and are ignored by crawlers; we use them to label each section, which makes the file readable when you revisit it months later. The file must be UTF-8 encoded without a byte-order mark — the generator outputs the correct encoding automatically.

Keep in mind that Disallow rules do not provide security: confidential URLs should be protected by authentication, not just hidden from crawlers. robots.txt is a search-engine politeness mechanism, not an access control system.