Free Robots.txt Generator

Generate a standards-compliant robots.txt in seconds — one-click templates for WordPress, Shopify, Blogger and Wix, plus full control over 23 search and AI crawlers.

Start with a template

One click to apply a recommended config. Tune anything below.

Allow All Let every crawler in — best for new sites WordPress Hide /wp-admin, /wp-includes, plugins, cache Shopify Hide /admin, /cart, /checkout, /account Blogger Hide /search and /p/preview duplicate URLs Wix Hide Wix internal APIs and partials Block AI Scrapers Opt out of 7 AI training bots — GPTBot, Claude, Gemini & more (search bots stay allowed)

AI Crawlers

AI training & answer-engine bots — block to keep your content out of LLM training and AI answers.

• GPTBot
• OAI-SearchBot
• ChatGPT-User
• ClaudeBot
• Claude-SearchBot
• PerplexityBot
• Perplexity-User
• Google-Extended
• Applebot-Extended
• Meta-ExternalAgent
• CCBot
• Bytespider

Search Engines

Major search crawlers — most sites want these allowed.

• Google
• Google Image
• Google Mobile
• Bingbot
• DuckDuckGo
• Yahoo
• Yandex

Regional Engines

Region-specific crawlers — only block if you don't serve those markets.

• Baidu
• Naver
• Sogou
• Seznam

Default — uses the All-Robots rule above Explicitly allowed Blocked

Restricted Directories

Add

Generate robots.txt Reset

Example output — click Generate to build yours

# EXAMPLE — what the generator will output
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /private/

Sitemap: https://example.com/sitemap.xml

How to Generate a robots.txt File in 3 Steps

1. Pick a template (optional but recommended). Click WordPress, Shopify, Blogger, Wix, Allow All, or Block AI Scrapers to load a battle-tested config. You can still fine-tune anything afterwards.
2. Adjust crawlers and directories. Open Customize to allow or block specific bots across three groups — Search Engines, AI Crawlers, Regional Engines — and add any directory paths you want kept out of search results (e.g. /admin, /private).
3. Click Generate robots.txt. Copy the output or download it as robots.txt, then upload it to your site root (https://yoursite.com/robots.txt).

What Is robots.txt and Why Every Site Needs One in 2026

robots.txt is a plain-text file at the root of your domain that tells automated crawlers which parts of your site they're allowed to fetch. It uses a tiny set of directives — User-agent, Allow, Disallow, Crawl-delay, Sitemap — and every well-behaved crawler reads it before doing anything else.

A correctly configured robots.txt does three jobs at once:

• Tells search engines what to crawl
• Points crawlers to your Sitemap
• Gives you control over AI crawlers, on whichever side you want.

Two more things worth knowing:

• robots.txt is a request, not a wall.
• Blocking in robots.txt is not the same as blocking from search results.

Common robots.txt Mistakes That Hurt SEO

We see the same five mistakes again and again in audits:

• Disallowing your entire site by accident.
• Blocking CSS or JavaScript directories.
• Trying to use robots.txt for noindex.
• Forgetting the Sitemap line.
• Mixing up robots.txt blocking with security.

Use Cases

• Set up a clean robots.txt for a new WordPress, Shopify, Blogger or Wix site.
• Tell search engines where your Sitemap is.
• Stop AI scrapers from training on your content — but stay citable in AI answers.
• Slow down aggressive crawlers on a small server.
• Keep staging or dev environments out of the index.

FAQs

Where do I put the robots.txt file?

Upload it to the root of your domain so it resolves at https://yoursite.com/robots.txt. It must be at the root — not in a subdirectory. Each subdomain (blog.yoursite.com, shop.yoursite.com) needs its own separate robots.txt.

How do I block ChatGPT, Claude, and Perplexity from training on my content?

Apply the Block AI Scrapers template.

Is robots.txt legally binding?

No. robots.txt is a voluntary protocol.

Should every website have a robots.txt file?

Yes — even an empty or Allow: / file is better than a missing one.