Robots.txt Generator & Robots.txt Validator

A robots.txt file gives compliant crawlers instructions about which paths they may request. This tool helps you build those instructions and then analyze the live file from the same page. It is designed for site owners, developers, SEO teams, publishers, and marketers who need clear crawl controls without manually formatting every directive.

Use the generator when you are creating a new file or replacing an outdated one. Use the validator when you want a quick site-level view of the live robots.txt file, sitemap reference, broad crawler restrictions, crawlability, and detected recommendations.

1. Choose the crawler or User-agent. Apply a rule to all compliant crawlers with an asterisk, or select a specific crawler option available in the tool.
2. Set Allow and Disallow paths. Define the pages or directories that crawlers may request or should avoid. Paths are case-sensitive, so enter them exactly as they appear in your URLs.
3. Add your sitemap. Enter the complete sitemap URL, including the protocol and hostname for example, https://example.com/sitemap.xml.
4. Configure optional controls. Add Crawl-delay for crawlers that support it, choose AI-bot crawling controls, and add directory restrictions needed for your site.
5. Review the generated code. Check every path before publishing. A single sitewide Disallow rule can prevent compliant crawlers from accessing the whole site.
6. Copy or download the file. Copy the output to your clipboard or download the generated robots.txt file.
7. Upload it to the site root. Publish the file at https://yourdomain.com/robots.txt for the exact host it should control.
8. Run the validator. Enter the live URL after deployment to confirm that the file is accessible and review the reported status cards and recommendations.

The validator analyzes a live robots.txt URL rather than testing one individual page against a selected user agent. Enter your website or robots.txt address, select Check, and review the file-level results.

• Sitemap Found: Shows whether the fetched robots.txt file contains a detectable Sitemap directive.
• Admin Blocked: Highlights whether a broad rule appears to block crawler access across the site. Review any sitewide Disallow rule carefully before publishing.
• Crawlability: Provides a high-level assessment of how open or restrictive the detected rule set appears.
• ROBOTS.TXT tab: Displays the retrieved directives with line-level readability so you can inspect the live file.
• SITEMAP tab: Lets you review the sitemap information detected during the analysis.
• Issues & Recommendations: Summarizes positive checks and potential concerns, such as overall site crawlability and sitemap-directive presence.

Using Wildcards in Robots.txt Paths

Two pattern characters are supported by Google and most major crawlers, and both work in this generator:

Three rules that prevent the most common mistakes:

• A path with no wildcard is a prefix match. Disallow: /admin blocks /admin/, /admin-tools/, and /administrator/, not just the folder you meant. Add the trailing slash to limit it.

• When rules conflict, the most specific rule wins for Google, not the first one listed. A longer, more specific Allow overrides a shorter Disallow.

• Paths are case-sensitive. /Private/ and /private/ are different locations.

The right file depends on your site structure. This simple example allows general crawling, restricts two directories, and declares a sitemap:

Replace the example paths and domain with real values from your site. Do not publish a template unchanged, and do not list sensitive locations as a substitute for proper access control.

Where to Upload Robots.txt

Publish the file in the top-level directory of the host it controls. For example, rules for https://example.com/ must be available at:

A file at https://example.com/folder/robots.txt does not control the full domain. Rules also apply only to the matching host, protocol, and port. If you use separate subdomains, each one may need its own robots.txt file.

Need a sitemap URL to add to the file? Create one with the XML Sitemap Generator and paste the published sitemap location into the Sitemap field.

Robots.txt Controls Crawling, Not Guaranteed Indexing

Crawling and indexing are different stages. A Disallow rule can stop a compliant crawler from fetching a page, but the URL may still appear in search results when it is discovered through links or other sources. Robots.txt therefore should not be used as the only method for removing a page from search results.

To prevent indexing while allowing a search engine to access the page, use a supported noindex meta tag or X-Robots-Tag. To keep private content inaccessible to crawlers and users, use authentication, permissions, or another server-side control. Do not block a noindex page in robots.txt, because the crawler must fetch the page to see the noindex instruction.

AI companies use named crawlers to gather training data and to fetch pages when answering user questions. Blocking them in robots.txt is currently the standard way to signal that you do not want your content used. This generator includes controls for the ten AI crawlers that matter most in 2026:

Two things worth understanding before you block everything. 

Training crawlers and answer-engine crawlers do different jobs. Blocking GPTBot stops your content being used to train OpenAI's models. Blocking ChatGPT-User or OAI-SearchBot also stops your pages being retrieved and cited when someone asks a question, which removes you from AI answers entirely. If your goal is visibility in AI search, blocking every AI bot works against you.

Google-Extended is separate from Googlebot. Disallowing Google-Extended does not affect your normal Google Search rankings or crawling. It only controls whether your content feeds Gemini and Google's AI training, so you can block one without the other.

The limitation that applies to all of them: robots.txt is a voluntary protocol. Well-behaved crawlers respect it, but a non-compliant scraper can ignore it entirely. Use server rules, firewall rules, CDN bot management, or authentication when you need enforceable protection rather than a stated preference.

• Generate and analyze robots.txt from one tool instead of switching between separate pages.
• Configure standard crawl rules, sitemap locations, directory restrictions, Crawl-delay, and AI-bot controls.
• Review live sitemap detection, broad crawler-block status, and crawlability at a glance.
• Inspect the fetched robots.txt and sitemap information in dedicated result tabs.
• Copy content directly or download a .txt file for server upload.
• Add as many rules and restricted directories as your site needs. There is no rule limit.

Related SEO tools: generate an XML sitemap, create structured data, or build a clean SEO slug with other Online Tool Pot utilities.