Developer SEO Utility
robots.txt Generator
Generate robots.txt content by entering crawler access rules, allowed paths, blocked paths, and a Sitemap URL.
robots.txt Settings
Enter the site URL, crawler target, allowed paths, blocked paths, and Sitemap URL.
In general, allow public pages and block admin, search result, and private paths. However, robots.txt is not a security tool, so sensitive pages must be protected with server-side authentication and permissions.
Used to automatically create the Sitemap URL. Example: https://win-j.com
Use * to apply the rule to all search bots.
Delay between crawling requests. Only some search engines support this directive.
Enter one path per line. Example: /utility/
Enter paths you do not want search engines to crawl.
If left empty, /sitemap.xml will be generated based on the site URL.
Quick Options
Add commonly used robots.txt rules with simple checkboxes.
Generated Result
Save the content below as a robots.txt file in the site root path.
Enter settings and click Generate robots.txt.
Help
Where should robots.txt be placed?
The robots.txt file must be placed in the site root path. For example, it should be accessible as https://example.com/robots.txt.
Common Rules
- User-agent: * : Applies to all search bots
- Allow: / : Allows access to the entire site
- Disallow: /admin/ : Blocks admin pages
- Sitemap: https://example.com/sitemap.xml : Tells search engines where the Sitemap is located
Notes
robots.txt is only a crawling instruction for search engine crawlers, not a security mechanism. Sensitive admin pages or private data must be protected with server authentication and access control.
Will Disallow completely remove a page from search results?
robots.txt limits crawling. To remove already indexed pages, you may also need noindex, removal requests, canonical cleanup, or other SEO measures.
Do I have to add a Sitemap?
It is not required, but adding a Sitemap URL to robots.txt is recommended because it helps search engines understand your site structure.