🤖 Free Robots.txt Generator

Create a pristine Robots.txt file to optimize your SEO crawl budget. Safely instruct Googlebot what it should—and shouldn't—index.

Why Your Website Needs a Robots.txt File

Before Google indexes your website, its crawlers ("bots") first look for a file at `/robots.txt`. If you don't have this file configured correctly, you are leaving your technical SEO to chance.

  • Maximize Crawl Budget: Google spends a limited amount of time on your site. By using `Disallow` rules on admin folders (`/wp-admin/`) or internal searches, you force the bot to index your valuable money-pages.
  • Prevent Duplicate Content Penalties: If you use query parameters for tracking or filtering (e.g., `?sort=price`), robots.txt prevents search engines from indexing these variations as duplicate content.
  • Sitemap Discovery: Your `robots.txt` is the absolute best place to declare the location of your `sitemap.xml` file, ensuring search engines find all your links instantly.
  • Block Unwanted AI Scrapers: In 2026, you can explicitly use rules in your robots.txt to prevent generative AI companies from using your site's data to train their models without compensation.

Frequently Asked Questions

What is a robots.txt file?

It is a plain text file placed in your website's root directory that communicates with web crawlers (like Googlebot), telling them which parts of your site they should or shouldn't scan.

Where should I place the robots.txt file?

It must be placed in the highest-level root directory. For example, it must be accessible at exactly `https://www.yourdomain.com/robots.txt` to function correctly.

Can a robots.txt file hide my site from hackers?

No. Robots.txt is a public directive, not a security protocol. Malicious scrapers will ignore the rules entirely. Never put sensitive data paths in your robots.txt file—use authentication instead.

What the generator asks you

You choose the user agent the rules apply to (a specific crawler or * for all), add the paths to allow and the paths to disallow, and enter your sitemap URL. The generator assembles those into a correctly formatted robots.txt that you save at the root of your site. The company and website fields on the shared form are not used in this file.

What the finished document covers

  • User-agent line
  • Allow directives
  • Disallow directives
  • Sitemap directive

What it does not do

robots.txt is a request to well-behaved crawlers, not access control: it does not stop a page being indexed if other sites link to it, and it does not block scrapers that ignore the standard. Only one user-agent block is generated, so separate rules for different bots have to be added by hand. The file carries no legal weight and this page gives no legal advice.

How robots.txt works

robots.txt is a plain-text file at the root of your domain (https://example.com/robots.txt) that tells well-behaved crawlers which paths they may fetch. Each block starts with a User-agent line naming a crawler (or * for all), followed by Disallow and Allow rules matched against the URL path. A Sitemap: line pointing at your XML sitemap helps search engines discover pages faster.

  • It controls crawling, not indexing. A URL blocked in robots.txt can still appear in search results if other sites link to it. To keep a page out of the index, let it be crawled and add a noindex meta tag instead.
  • Don't block CSS and JavaScript. Google renders pages; blocking asset folders makes your pages look broken to it and can hurt rankings.
  • The longest matching rule wins. Allow: /blog/ beats Disallow: / for anything under /blog/. Use $ to anchor a pattern to the end of a URL and * as a wildcard.
  • It is public and advisory. Anyone can read it, and malicious bots ignore it. Never use it to "hide" admin panels — protect those with authentication.

Common configurations

Allow everything, declare the sitemap — the right default for most content sites: a single User-agent: * block with no Disallow rules and a Sitemap: line.

Keep crawlers out of internal search and cart pages — e-commerce sites often disallow /search, /cart and URLs with sorting parameters so crawl budget goes to product pages.

Opt out of AI training crawlers — separate User-agent blocks for GPTBot, CCBot, Google-Extended and similar let you decline AI training while still allowing normal search indexing. The generator includes these as a toggle.

After uploading the file, confirm it with Google Search Console's robots.txt report. A syntax error there usually means a stray blank line between a User-agent and its rules, or a rule written with a full URL instead of a path.

Documents you may need next

Guides on this topic