Crawl access
Robots.txt Generator
Decide exactly who may crawl what. Build user-agent groups, allow and disallow paths, crawl delay, and sitemap directives — then copy a valid robots.txt in your browser.
Point search engines to your sitemap
# robots.txt generated by BoostLogik # Generated on: 2026-11-05 User-agent: * Disallow: /admin/ Disallow: /private/ Disallow: /api/ Allow: /
1. Download or copy the file
Use the buttons above to get your robots.txt content.
2. Upload to your site root
Place the file at https://yoursite.com/robots.txt
3. Test your robots.txt
Use Google Search Console's robots.txt Tester to verify it works correctly.
💡 Pro Tips:
- Order matters: More specific rules should come first
- Use wildcards:
*matches any sequence - End with
$to match exact URLs - Test changes before deploying to production
Click to add a rule blocking this bot
How this helps
- Start from presets for common setups, then fine-tune each user-agent's allow and disallow paths.
- Control search and AI crawlers — including GPTBot, ClaudeBot, PerplexityBot, and Google-Extended.
- Add Sitemap and Host directives so crawlers can find your canonical URL set immediately.
- Copy or download a clean robots.txt and upload it to the root of your domain.
Frequently asked questions
What is robots.txt?
robots.txt is a plain-text file at the root of a domain that tells crawlers which paths they may or may not fetch. It is a convention followed by well-behaved bots — including search engines and AI crawlers — and it is not a security mechanism.
Can I use robots.txt to block AI crawlers?
Yes. You can add user-agent groups for GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and others. Blocking a crawler in robots.txt is a request; reputable AI crawlers honor it, though it does not stop scrapers that ignore the convention.
Where does robots.txt go?
Place the file at the root of your domain so it is served at https://example.com/robots.txt — exactly that path, with no subdirectory or trailing filename.
What does Allow and Disallow mean?
Disallow blocks a crawler from a path; Allow re-permits a path inside a broader disallow. When rules conflict, the most specific path wins, so ordering within a group matters less than specificity.
Is this robots.txt generator free?
Yes. It runs entirely in your browser, requires no account, and you can generate as many configurations as you need.
Is anything important blocked by accident?
Run a free SEO health check to confirm your robots.txt, sitemap, and canonical signals let crawlers reach the pages that matter.