Robots.txt Generator
Build a valid robots.txt file for crawlers.
Add the website, paths, and crawler rules you need. The tool creates a file you can review and copy.
Use the tool
Enter the domain and only the crawl rules you understand and intend to use.
- 01
Enter the website domain and sitemap address.
- 02
Add one allow or disallow path per line.
- 03
Choose whether to block the listed AI crawlers.
- 04
Review the complete file before adding it to the website root.
What the result means
Robots.txt controls crawling, not indexing by itself.
An incorrect rule can hide important pages from crawlers. Review the generated file with your developer, especially before changing rules on an active ecommerce or lead-generation website.
Have Enok review these crawl rulesUse it to
- Creates allow and disallow rules
- Adds a sitemap reference
- Supports common AI crawler choices
Scope of this check
Know what the result can tell you.
What it checks
- Copy-ready robots.txt file
- Rule counts
- Crawler and sitemap directives
What it cannot prove
Review crawl rules before publishing them.
- Robots.txt controls crawler access but does not reliably remove a URL from search.
- A single broad rule can block important product, service, or content pages.
- Not every crawler supports crawl delay or follows every directive.
Questions before you act
Robots.txt Generator questions
01Where does the robots.txt file go?
Publish it at the root of the hostname, such as example.com/robots.txt, so crawlers can find it before requesting other pages.
02Can robots.txt remove a page from Google?
No. Blocking crawling can prevent Google from seeing a noindex directive and the URL may remain known. Use the appropriate indexing and removal method for the situation.
03Should I block AI crawlers?
That is a business and content-licensing decision. Blocking a crawler can also reduce how often the brand or content appears in that system.