# Free Robots.txt Generator

Generate a standards-compliant robots.txt in seconds — one-click templates for WordPress, Shopify, Blogger and Wix, plus full control over 23 search and AI crawlers.

## Start with a template

One click to apply a recommended config. Tune anything below.

Allow All Let every crawler in — best for new sites WordPress Hide /wp-admin, /wp-includes, plugins, cache Shopify Hide /admin, /cart, /checkout, /account Blogger Hide /search and /p/preview duplicate URLs Wix Hide Wix internal APIs and partials Block AI Scrapers Opt out of 7 AI training bots — GPTBot, Claude, Gemini & more (search bots stay allowed)

### AI Crawlers
AI training & answer-engine bots — block to keep your content out of LLM training and AI answers.
- GPTBot
- OAI-SearchBot
- ChatGPT-User
- ClaudeBot
- Claude-SearchBot
- PerplexityBot
- Perplexity-User
- Google-Extended
- Applebot-Extended
- Meta-ExternalAgent
- CCBot
- Bytespider

### Search Engines
Major search crawlers — most sites want these allowed.
- Google
- Google Image
- Google Mobile
- Bingbot
- DuckDuckGo
- Yahoo
- Yandex

### Regional Engines
Region-specific crawlers — only block if you don't serve those markets.
- Baidu
- Naver
- Sogou
- Seznam

Default — uses the All-Robots rule above Explicitly allowed Blocked

### Restricted Directories
Add

Generate robots.txt Reset

Example output — click Generate to build yours

```
# EXAMPLE — what the generator will output
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /private/

Sitemap: https://example.com/sitemap.xml
```

## How to Generate a robots.txt File in 3 Steps

1. **Pick a template** (optional but recommended). Click _WordPress_, _Shopify_, _Blogger_, _Wix_, _Allow All_, or _Block AI Scrapers_ to load a battle-tested config. You can still fine-tune anything afterwards.
2. **Adjust crawlers and directories**. Open _Customize_ to allow or block specific bots across three groups — _Search Engines_, _AI Crawlers_, _Regional Engines_ — and add any directory paths you want kept out of search results (e.g. `/admin`, `/private`).
3. **Click Generate robots.txt**. Copy the output or download it as `robots.txt`, then upload it to your site root (`https://yoursite.com/robots.txt`).

## What Is robots.txt and Why Every Site Needs One in 2026

`robots.txt` is a plain-text file at the root of your domain that tells automated crawlers which parts of your site they're allowed to fetch. It uses a tiny set of directives — `User-agent`, `Allow`, `Disallow`, `Crawl-delay`, `Sitemap` — and every well-behaved crawler reads it before doing anything else.

A correctly configured `robots.txt` does three jobs at once:
- **Tells search engines what to crawl**
- **Points crawlers to your Sitemap**
- **Gives you control over AI crawlers, on whichever side you want.**

Two more things worth knowing:
- **`robots.txt` is a request, not a wall.** 
- **Blocking in `robots.txt` is not the same as blocking from search results.**

## Common robots.txt Mistakes That Hurt SEO
We see the same five mistakes again and again in audits:
- **Disallowing your entire site by accident.** 
- **Blocking CSS or JavaScript directories.** 
- **Trying to use `robots.txt` for noindex.** 
- **Forgetting the Sitemap line.** 
- **Mixing up `robots.txt` blocking with security.**

## Use Cases
- **Set up a clean robots.txt for a new WordPress, Shopify, Blogger or Wix site.** 
- **Tell search engines where your Sitemap is.** 
- **Stop AI scrapers from training on your content — but stay citable in AI answers.** 
- **Slow down aggressive crawlers on a small server.** 
- **Keep staging or dev environments out of the index.**

## FAQs
### Where do I put the robots.txt file?
Upload it to the root of your domain so it resolves at `https://yoursite.com/robots.txt`. It must be at the root — not in a subdirectory. Each subdomain (`blog.yoursite.com`, `shop.yoursite.com`) needs its own separate `robots.txt`.
### How do I block ChatGPT, Claude, and Perplexity from training on my content?
Apply the _Block AI Scrapers_ template. 
### Is robots.txt legally binding?
No. `robots.txt` is a voluntary protocol. 
### Should every website have a robots.txt file?
Yes — even an empty or `Allow: /` file is better than a missing one.