Free Robots.txt Generator and Tester
A single misplaced Disallow line can wipe an entire section of a site out of Google. This free tool builds a syntactically valid robots.txt from simple form fields — one rule group per crawler, wildcard support, crawl-delay, and a sitemap directive — then lets you test any URL path against those rules to confirm it is crawlable before you deploy.
Robots.txt Generator
Build a valid robots.txt with one rule group per crawler, then download or paste it into your site root.
Generated robots.txt
User-agent: * Disallow: /app/ Disallow: /auth/ Allow: / Sitemap: https://rankthrottle.com/sitemap.xml
Robots.txt Tester
Check whether a specific URL path is crawlable under the rules above.
Blocked
Blocked by Disallow: /app/
What this tool checks
- Multiple user-agent rule groups with Allow and Disallow paths
- Wildcard (*) and end-of-URL ($) pattern support
- Sitemap directive and optional crawl-delay
- Path tester showing which rule wins and why
Frequently asked questions
Does robots.txt remove a page from Google?
No. Disallow blocks crawling, not indexing — a blocked URL can still appear in results without a description. To remove a page from the index, allow crawling and use a noindex robots meta tag instead.
Where does robots.txt need to live?
At the root of the domain, for example https://example.com/robots.txt. Crawlers do not look for it in subfolders, and each subdomain needs its own file.
More free SEO tools
Want SEO data backed by real Google rankings?
RankThrottle connects to Google Search Console to surface true ranking opportunities, keyword gaps versus competitors, and AI-powered content recommendations.