# Free Robots.txt Generator — Foreground Digital

> Build a valid robots.txt in under a minute. With a plain-English explainer of User-agent, Allow, Disallow, Sitemap, and the rules that actually matter for SEO.

Source: https://foreground.agency/tools/robots-txt-generator/

---

[Free tools](https://foreground.agency/tools) / Robots.txt Generator 

# 
Robots.txt, demystified.And then generated.

 

Most "free generators" hand you a file before you understand what it does. We flipped it. Read the short explainer below, then use the builder — you'll know exactly what you're shipping.

 How to write robots.txt 

## 
Five things to know before you write a single line.

 01 — What it is 

### A polite request, not a security boundary.

 

 robots.txt sits at the root of your domain (e.g. foreground.agency/robots.txt). It tells well-behaved crawlers — Googlebot, Bingbot, GPTBot — which paths they should and shouldn't fetch. It is not access control. Anyone can ignore it. Don't put secrets in Disallow — you'll just signpost them to attackers.

 02 — The four directives that matter 

### User-agent, Disallow, Allow, Sitemap.

 User-agent: 

Which crawler does the next block apply to. * means "all bots." Specific names: Googlebot, Bingbot, GPTBot, ClaudeBot.

 Disallow: 

Path the bot should not crawl. Always starts with /. Disallow: / blocks the whole site; Disallow: (empty) blocks nothing.

 Allow: 

Carves an exception inside a Disallow. Useful when blocking a folder but keeping one file inside it crawlable. More-specific rules win.

 Sitemap: 

Absolute URL of your sitemap.xml. Sits outside any user-agent block. You can list more than one. Most crawlers also auto-discover at /sitemap.xml.

 03 — A real example 

### What a sane default looks like.

 

# Default policy — every crawler
User-agent: *
Disallow: /admin/
Disallow: /api/
Disallow: /cart
Disallow: /checkout
Allow: /api/og-image/

# Block AI training bots specifically (optional)
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

# Always include the sitemap
Sitemap: https://example.com/sitemap.xml 
Read top-down: the User-agent: * block applies to everyone. Then GPTBot and ClaudeBot get their own block that overrides the default — they get Disallow: /, blocking the entire site. Sitemap lives at the bottom outside any block.

 04 — Wildcards + edge cases 

### Two wildcards. That's it.

 
 - ·* — match any sequence of characters. Disallow: /*?ref= blocks any URL with ?ref= in it.
 - ·$ — anchor to end of URL. Disallow: /*.pdf$ blocks PDF files but not /foo.pdf?download=1.
 - ·Lines are case-sensitive on path, case-insensitive on directives. /Admin ≠ /admin.
 - ·Comments start with #. Use them — your future self will thank you.
 
 05 — Common mistakes 

### The four ways people break their SEO.

 
 - ·Disallowing CSS or JS. Googlebot needs to fetch your stylesheets and scripts to render the page. Don't Disallow: /static/.
 - ·Using robots.txt to hide pages from search results. It only blocks crawling — a Disallowed URL with external backlinks can still appear in SERPs as a title-only result. Use noindex meta tag for that.
 - ·Forgetting trailing slashes matter. Disallow: /blog blocks /blog, /blog/, AND /blog-posts. Use /blog/ if you mean the folder.
 - ·Not testing it. Use Google Search Console's robots.txt Tester (or paste your file into our preview below) before you ship.
 
 Generator 

## 
Now build it.

 

Pick a preset, tweak the paths, paste your sitemap URL. The preview updates as you type.

 Preset Open — allow all crawlers (recommended for marketing sites) Open to search, block AI training bots (GPTBot, ClaudeBot, CCBot, Google-Extended) Staging — block everyone Custom — start from blank Sitemap URL (optional but recommended) Disallow paths (one per line) Allow paths (optional — overrides Disallow) Crawl delay in seconds (optional, ignored by Googlebot) Preview · robots.txt Copy Download 

 
Save as robots.txt at the root of your domain (same folder as index.html).

 
Optional
 

### Found this useful? Share it — keeps the tool free + updated. 

 

No login, no email gate. We just like knowing it’s being used.

 X FB LI WA TG Copy Pair it with

## 
Other free tools that fit the same workflow.

[See all tools →](https://foreground.agency/tools)[Free tool

### LLMs.txt File Generator

Tell ChatGPT, Claude, and Perplexity what your site is. Builds the new Answer.AI standard llms.txt.

Open tool →](https://foreground.agency/tools/llms-txt-generator)[Free tool

### 2026 SEO Audit

Crawls up to 25 pages — on-page, technical, AI search readiness, structure, authority. Scored out of 100. Emailed PDF.

Open tool →](https://foreground.agency/tools/seo-audit-2026) Need the full picture? 

## 
Robots.txt is one line of a real SEO audit.

 

We'll audit your crawl directives, sitemap structure, internal linking, Core Web Vitals, schema markup, and the gap between what Google sees and what you want it to see.

 [Get an audit →](https://foreground.agency/contact)