robots.txt generator
robots.txt generator
Build a robots.txt from the paths you want crawled and the ones you do not, with your sitemap declared and the rules ordered so a specific allowance is not swallowed by a broad block.
How this is calculated
One user-agent group is written, with Allow lines before Disallow lines so the more specific rule is read first, then Crawl-delay if set, then the Sitemap line outside the group. Paths are kept exactly as entered after trimming whitespace.
- robots.txt is not access control. It asks compliant crawlers not to fetch a path; it does not stop a hostile client, and a blocked URL can still be indexed if other sites link to it. Anything genuinely private needs authentication, not a Disallow line.
- The file has to sit at the root of the host — https://example.com/robots.txt — and it governs only that host and scheme. A file in a subdirectory is not read, and a separate subdomain needs its own.
- Blocking a CSS or JavaScript file that a page needs to render can prevent a crawler from understanding the page. Allow the assets a rendered view depends on even when the directory around them is blocked.
- Matching is by prefix, and the longest matching rule wins for the crawlers that implement it that way. A Disallow of /tools/ also matches /tools-landing/, because the match is on the string and not on a directory boundary.
robots.txt is a request to crawlers, not a lock on a door. It is worth writing carefully because a mistake in it is site-wide and silent — a single Disallow line can remove every page from a crawler’s queue without anything on the site looking wrong.
What the file can and cannot do
It asks compliant crawlers not to fetch a path. It does not stop a hostile client, and it does not prevent indexing: a blocked URL can still appear in a search result if other sites link to it, because the crawler knows the address without having read the page. Anything genuinely private needs authentication.
The file must sit at the root of the host and governs only that host and scheme. A robots.txt in a subdirectory is not read, and a subdomain needs its own. Blocking a stylesheet or script that a page needs to render can also prevent a crawler from understanding the page, so allow the assets a rendered view depends on even inside a blocked directory.
Matching is on strings, not directories
A Disallow of /tools/ also matches /tools-landing/, because the match is a prefix of the path string rather than a directory boundary. That surprises people who read the trailing slash as a folder marker. Where a crawler implements longest-match precedence, a more specific Allow rescues the exception — which is why Allow lines are written before Disallow lines here.
Crawl-delay is honoured by some crawlers and ignored by others, so it is not a rate limit you can rely on. Leave it blank unless a crawler has asked you for one, and use server-side controls for anything that genuinely needs protecting.
Questions readers ask
Does an empty Disallow mean allow everything?
Yes. “Disallow:” with no path is the explicit way to say nothing is blocked, and it is what this generator writes when you supply no rules at all.
Should I block my admin directory?
Blocking it saves crawl effort, but the directory should be protected by authentication regardless. A Disallow line is not what keeps an admin area private, and treating it as the protection is the mistake.
More in Markup, feeds and tagged links
SEO and website tools
Titles and snippets measured, copy counted and scored, and the markup a page needs built in the browser and copied out.
- robots.txt generator Build a robots.txt from the paths you want crawled and the ones you do not, with your sitemap declared and the rules ordered so a specific allowance is not swallowed by a broad block.
- XML sitemap generator Turn a list of URLs into a valid urlset sitemap, with duplicates removed, XML escaping applied and the protocol limits checked against what you pasted.
- Schema markup generator Build JSON-LD for a website, an organisation, an article or a breadcrumb trail from fields you fill in, with the properties each type requires present before the block is written.
- FAQ schema generator Turn question and answer pairs into FAQPage JSON-LD, with the answers kept as the text a reader can already see on the page.
- Hreflang tag generator Build the alternate-language link set for a page, including the x-default entry, from the language and region codes and URLs you supply.
- Open Graph and Twitter card generator Build the Open Graph and Twitter card meta tags for a page, with the image dimensions you supply deciding which card type is honest to claim.
- UTM link builder Add campaign parameters to a URL with the values encoded properly, keeping any query string and fragment already on the link intact.