Robots.txt Generator
Set allow/disallow rules per user-agent and add a sitemap reference to generate a ready-to-use robots.txt file — the standard way to guide search engine crawlers around your site.
SEO & Marketing Tools Runs entirely in your browser
The file is generated entirely on your device — nothing is sent to a server.
Generated robots.txt
User-agent: * Allow: / Disallow: /admin Disallow: /api/
What this tool does
Builds a robots.txt from paths you allow and disallow plus an optional sitemap URL, ready to download.
When to use it
- Launching a site and needing a crawler directive file.
- Keeping crawlers out of administrative or internal paths.
- Adding your sitemap URL so crawlers find it.
- Fixing an existing file by rebuilding it properly.
How to use it
- Enter the "Allow" paths, one per line.
- Enter the "Disallow" paths, one per line.
- Enter the "Sitemap URL" if you have one.
- Copy the "Generated robots.txt" or click "Download".
A practical example
Input
Disallowing /admin/ with a sitemap URL.
Output
A file containing that disallow directive and the sitemap line, ready for the site root.
Limitations
- Blocking crawling is not blocking indexing: a page disallowed in robots.txt can still appear in results if other sites link to it. Use a noindex tag to prevent indexing — and the two conflict, because a blocked crawler never reads the tag.
- This is not a security measure: the file is public, and naming a sensitive path in it points at that path rather than hiding it.
- Well-behaved crawlers respect it; badly behaved ones ignore it entirely.
- The file must sit at the domain root exactly to be read at all.
Privacy and processing
The file is generated entirely on your device and nothing is sent to a server.
Technical notes
- This generates the file; it does not upload it or verify where it sits on your server.