robots.txt generator

robots.txt controls crawling, not indexing

The single most common mistake: people Disallow a page to keep it out of Google. That's backwards. Disallow stops crawling — but a disallowed URL can still get indexed (without a snippet) if it's linked elsewhere. To keep a page out of the index, let it be crawled and add a noindex meta tag. robots.txt is for crawl budget and access, not de-indexing.

The directives, in plain English

A robots.txt file is just a list of groups. Each group names a User-agent and then gives it rules. Three directives do almost all the work:

• Disallow — blocks a path from being crawled. Disallow: /admin keeps bots out of everything under /admin. This is the "robots txt disallow" rule everyone looks for.
• Allow — carves an exception out of a broader Disallow. Useful when you block a folder but want one file inside it crawled.
• Sitemap — points crawlers at your sitemap so they discover your URLs faster. Add a Sitemap: line with the full URL; you can list more than one.

Two patterns come up constantly:

• Allow all — let every bot crawl everything. That's User-agent: * followed by Disallow: (empty). An empty Disallow means "nothing is off-limits." A blank or missing robots.txt does the same thing.
• Disallow all — block every bot from the whole site. That's User-agent: * followed by Disallow: /. The single slash matters — it means "the entire site." Use it on staging, never by accident on production.

WordPress robots.txt

WordPress serves a virtual robots.txt by default — there's no physical file in your folder, WordPress generates one on the fly at /robots.txt. It usually disallows /wp-admin/ and allows /wp-admin/admin-ajax.php. If you upload a real robots.txt to the site root, that physical file takes over and the virtual one stops. SEO plugins like Yoast or Rank Math let you edit the virtual version from the dashboard. Either way, build the rules here, then paste them in.

Generator vs. validator vs. tester

This is a robots.txt generator — it writes valid syntax so you don't fat-finger a directive. It is not a live validator. The workflow: build the file here, upload it to your root, then test it against real URLs. Google Search Console has a robots.txt report that flags syntax errors and shows whether a given URL is blocked. For your own quick checks, open /robots.txt on the live domain and confirm it serves the rules you expect. A generator that produces clean syntax plus a quick test pass beats hand-editing every time.

Blocking AI crawlers

Major AI crawlers honor robots.txt. If you don't want your content training models or feeding AI answers, the AI-bot toggle adds a group disallowing GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot and more. Want to check what your current robots.txt allows? Run it through the AI-bot access checker. While you're tidying up crawl signals, the meta & OG checker and SERP preview cover the on-page tags that decide how those crawled pages actually show up.

Getting crawl rules, indexing, and the technical layer right across a large site is its own job. That's the programmatic SEO and fractional SEO work we do — or start with a free SEO audit if you want a second set of eyes first.