Crawl access

Robots.txt Generator

Decide exactly who may crawl what. Build user-agent groups, allow and disallow paths, crawl delay, and sitemap directives — then copy a valid robots.txt in your browser.

Quick Presets
Start with a common configuration
Crawl Rules
Define which bots can access which paths
Rule 1
Additional Directives

Point search engines to your sitemap

Generated robots.txt
Copy and save as robots.txt in your root directory
# robots.txt generated by BoostLogik
# Generated on: 2026-11-05

User-agent: *
Disallow: /admin/
Disallow: /private/
Disallow: /api/
Allow: /
How to Use

1. Download or copy the file

Use the buttons above to get your robots.txt content.

2. Upload to your site root

Place the file at https://yoursite.com/robots.txt

3. Test your robots.txt

Use Google Search Console's robots.txt Tester to verify it works correctly.

💡 Pro Tips:

  • Order matters: More specific rules should come first
  • Use wildcards: * matches any sequence
  • End with $ to match exact URLs
  • Test changes before deploying to production
Block AI Crawlers
Common AI bots you may want to block
+ GPTBot+ OAI-SearchBot+ ChatGPT-User+ CCBot+ anthropic-ai+ ClaudeBot+ Google-Extended

Click to add a rule blocking this bot

How this helps

  • Start from presets for common setups, then fine-tune each user-agent's allow and disallow paths.
  • Control search and AI crawlers — including GPTBot, ClaudeBot, PerplexityBot, and Google-Extended.
  • Add Sitemap and Host directives so crawlers can find your canonical URL set immediately.
  • Copy or download a clean robots.txt and upload it to the root of your domain.

Frequently asked questions

What is robots.txt?

robots.txt is a plain-text file at the root of a domain that tells crawlers which paths they may or may not fetch. It is a convention followed by well-behaved bots — including search engines and AI crawlers — and it is not a security mechanism.

Can I use robots.txt to block AI crawlers?

Yes. You can add user-agent groups for GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and others. Blocking a crawler in robots.txt is a request; reputable AI crawlers honor it, though it does not stop scrapers that ignore the convention.

Where does robots.txt go?

Place the file at the root of your domain so it is served at https://example.com/robots.txt — exactly that path, with no subdirectory or trailing filename.

What does Allow and Disallow mean?

Disallow blocks a crawler from a path; Allow re-permits a path inside a broader disallow. When rules conflict, the most specific path wins, so ordering within a group matters less than specificity.

Is this robots.txt generator free?

Yes. It runs entirely in your browser, requires no account, and you can generate as many configurations as you need.

Is anything important blocked by accident?

Run a free SEO health check to confirm your robots.txt, sitemap, and canonical signals let crawlers reach the pages that matter.