Who this helps
- Site owners preparing crawl rules
- Developers reviewing a robots.txt change
Developer SEO Utility
robots.txt Generator
Generate robots.txt content by entering crawler access rules, allowed paths, blocked paths, and a Sitemap URL.
Create Crawling Rules
Easily configure User-agent, Allow, and Disallow rules.
Connect Sitemap
Add a sitemap.xml URL to help search engines discover your site structure.
Copy and Download
Copy the generated robots.txt content or save it as a file.
User-agent
*
Allow
0 items
Disallow
0 items
Sitemap
Enabled
robots.txt Settings
Enter the site URL, crawler target, allowed paths, blocked paths, and Sitemap URL.
Recommended Structure
In general, allow public pages and block admin, search result, and private paths. However, robots.txt is not a security tool, so sensitive pages must be protected with server-side authentication and permissions.
Used to automatically create the Sitemap URL. Example: https://win-j.com
Use * to apply the rule to all search bots.
An optional directive supported by some search engines. Googlebot does not support Crawl-delay.
Enter one path per line. Example: /utility/
Enter paths you do not want search engines to crawl.
If left empty, /sitemap.xml will be generated based on the site URL.
Quick Options
Add commonly used robots.txt rules with simple checkboxes.
Generated Result
Save the content below as a robots.txt file in the site root path.
Enter settings and click Generate robots.txt.
Building robots.txt rules with a clear crawling purpose
Generate User-agent, Allow, Disallow and optional Sitemap lines for a site's robots.txt. These are crawling instructions, so review them separately from access control and indexing decisions.
When to use this tool
- Group administrative and search paths for review.
- Add the location of an existing sitemap.
From input to a checked result
- 1
List the public paths and paths you want crawlers to avoid.
- 2
Enter * or the intended crawler product token.
- 3
Enter one Allow or Disallow path per line.
- 4
Add an absolute sitemap URL or a site address for the default /sitemap.xml.
- 5
Generate, review and place the file at the site's /robots.txt.
- 6
Verify the deployed file and the target crawler's behavior.
Avoid admin crawling
- Input
- User-agent: *; Disallow path: /admin/
- Result
- A User-agent group containing Disallow: /admin/
This rule does not protect the admin page from visitors.
Advertise a sitemap
- Input
- Site: https://example.com; Sitemap enabled
- Result
- Sitemap: https://example.com/sitemap.xml
Check that the file actually exists before deployment.
How it works
The tool validates the agent token, accepts up to 10,000 entered rule lines and removes duplicate paths within each list. It checks HTTP(S) URL syntax without credentials. Optional Crawl-delay accepts a finite nonnegative decimal.
Limitations
- Rules are not tested against a live server or crawler.
- Googlebot does not support Crawl-delay.
- Disallow is not a reliable method for removing a URL from search results.
- The file is public and is not a security boundary.
Your data
Rules are assembled in the browser without saving them to win-j. Review path names before publishing the public file.
References
Related tools
- sitemap.xml Generator
Convert a URL list into XML, resolving relative paths and removing duplicates. Select canonical, indexable pages before generating the file; the tool cannot verify their live status.
- Canonical URL Checker
Find missing, duplicate and questionable canonical links before publishing. The checker reads supplied HTML; it cannot confirm which canonical URL a search engine has selected.
Frequently asked questions
Will Disallow remove a page from search?
It controls crawling, not guaranteed removal. Choose an indexing or access-control method appropriate to the page.
Does Crawl-delay work for Googlebot?
No. Check the documentation of any other crawler you intend to target.
Why was my delay rejected?
Use a nonnegative decimal such as 2 or 0.5, without exponent notation, text or an infinite value.