🤖 Robots.txt Generator

What robots.txt Actually Does (And What It Doesn't)

There's a persistent myth that robots.txt "blocks" pages from search engines. It doesn't — not reliably, anyway. What robots.txt does is issue instructions to well-behaved crawlers about which pages you'd prefer they skip. Google, Bing, Yandex, and most major bots follow these instructions voluntarily. A malicious scraper? It'll ignore your file entirely. Understanding this distinction is the first step to writing a robots.txt that actually serves your site well.

Every web server should have a robots.txt at its root — meaning https://yoursite.com/robots.txt. Bots check this URL before crawling anything else. If they find no file, most assume they have full access to everything. That's fine for a simple blog, but for an e-commerce site with hundreds of internal search result pages, empty cart URLs, and admin panels, you want precise control.

Understanding the Syntax: User-agent, Disallow, Allow

The file is read top to bottom, one "block" at a time. Each block starts with a User-agent: directive that names which crawler the following rules apply to. The wildcard * catches all bots; a specific name like Googlebot targets only Google's crawler.

After the User-agent line come your Disallow: and Allow: directives. A bare Disallow: with nothing after it means "crawl everything" — it's the polite way to tell bots they're welcome everywhere. Disallow: / is the nuclear option: no crawling at all. Most sites live somewhere between those two extremes.

One subtlety that trips people up: Allow: takes precedence over Disallow: when both match a URL and the Allow path is longer or equally long. So if you've blocked /wp-admin/ but want bots to access your admin's theme CSS file, you can write Allow: /wp-admin/admin-ajax.php before the broader Disallow. The more specific rule wins.

Which Paths Should You Actually Disallow?

A good robots.txt isn't about hiding content — Google can still index disallowed pages if they're linked from elsewhere; it just won't read their content. The real reason to disallow certain paths is to save your crawl budget.

Crawl budget is the rough number of pages a search engine bot will crawl on your site within a given timeframe. For small sites this doesn't matter much, but for large sites with thousands of pages, you want bots spending their time on your valuable content, not internal search pages, filtered product listings, or thank-you pages nobody should be ranking for.

Common sense candidates for disallowing include: /cart, /checkout, /account, /search (and its query-string variants), /admin/, /wp-admin/, /cgi-bin/, /tmp/, and any staging or internal tools. None of these need to rank in Google. They're just noise in the crawl queue.

On the other hand, be careful not to disallow paths that your important pages depend on. A classic mistake is blocking /wp-content/ or JavaScript and CSS directories. When Google can't render your pages because the assets are blocked, it sees a broken site — which affects rankings. If you're using a WordPress or other CMS, always verify that your theme's critical assets remain crawlable.

Crawl-Delay: Use It Carefully

The Crawl-delay: directive tells a bot how many seconds to wait between requests. This can help if aggressive crawling is putting strain on your server. A value of 10 means the bot should wait 10 seconds between each page request.

Here's the catch: Googlebot completely ignores Crawl-delay. If you're trying to throttle Google, you need to use the crawl rate settings inside Google Search Console instead. Bing and Yandex do honor the directive, so it's not useless — just understand its limited scope before relying on it.

Don't set Crawl-delay unless you have a genuine server load problem. An unnecessarily large delay could slow down indexing of your new content, which is rarely what site owners want.

The Sitemap Directive: Simple But Underused

Adding Sitemap: https://yoursite.com/sitemap.xml at the end of your robots.txt is one of the lowest-effort, highest-value SEO moves you can make. Bots that read robots.txt will immediately discover your sitemap URL and use it to find all your important pages — even ones with few or no inbound links.

You can include multiple Sitemap lines if you have separate sitemaps for images, videos, or different site sections. Each one gets its own line. The sitemap directive doesn't belong inside a User-agent block; it's a global directive that goes at the file level, typically at the very end.

One thing to verify: your sitemap URL must be absolute (include the full https:// prefix), and it should return a valid XML sitemap — not a redirect. If your sitemap is generated dynamically by a plugin or CMS, double-check that the URL actually works before putting it in robots.txt.

Per-Bot Rules: When Generic Isn't Enough

Most sites only need a single wildcard block. But there are situations where you want different rules for different bots. Maybe you're okay with Google crawling your /api/ endpoints for structured data purposes, but you don't want Bingbot wasting crawl budget on them. Or you want to allow Googlebot access to pages you're blocking everyone else from seeing during a soft launch.

When you target a specific bot by name, those rules override the wildcard block for that bot entirely. So if you have User-agent: * with Disallow: /secret/ and then a separate User-agent: Googlebot block with no Disallow, Google can access /secret/ while everyone else is blocked. Keep this specificity in mind when auditing an existing robots.txt — the interactions between named blocks and the wildcard can be non-obvious.

Testing Before You Deploy

An incorrectly written robots.txt can accidentally block your entire site from being indexed — and you might not notice for weeks until your traffic drops. Before uploading, run your file through Google Search Console's robots.txt tester (found under Settings → robots.txt). It lets you test specific URLs against your file and see exactly which rule is applied.

Also check: the file must be served as plain text (text/plain content-type), must be UTF-8 encoded with no BOM, and must live at the root of the domain. A robots.txt at https://example.com/blog/robots.txt won't be read by anything — it must be at https://example.com/robots.txt.

After deploying, wait for Google to re-crawl the file — usually within a day or two. Then audit your Search Console coverage report to see if any pages you intended to keep indexable are now showing as "Excluded by robots.txt." That's your signal to refine the rules and redeploy.