Ventora
PricingBlogCasesAbout
Log inGet started
  1. Ventora
  2. Free tools
  3. Free Robots.txt Generator, Tester & Sitemap Tools
Marketing

Free Robots.txt Generator, Tester & Sitemap Tools

Build robots.txt with AI crawler rules, test any URL with Google’s matching rules, and generate or validate sitemap.xml. Free, no signup, runs in your browser.

  • 100% free
  • No signup
  • No limits
  • Runs in your browser
Your site

Used for the Sitemap line. Each subdomain needs its own robots.txt.

Start from

Keeps admin, account and internal search result pages out of the crawl.

AI crawlers

Blocks crawlers that collect training data; AI search and user fetches still see your site, so you can be cited.

One path per line, e.g. /private/ or /*.pdf$

Exceptions inside a blocked folder.

Leave empty for https://www.example.com/sitemap.xml

Optional. Bing and Yandex read it; Google ignores it.

robots.txt
# Rules for all crawlers
User-agent: *
Disallow: /admin/
Disallow: /account/
Disallow: /search$
Disallow: /search?

# AI training crawlers: blocked (AI search can still cite you)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

The Sitemap line points at example.com. Type your site address under “Your site” before you copy or download.

Where it goes

Upload it as robots.txt to the root of your site so it opens at https://www.example.com/robots.txt. Google usually picks up changes within a day.

Need the website itself?

Describe your business and Ventora builds the site, the landing page and the checkout.

200 free credits, no card.

Next step · Website GraderGrade the whole site

A robots.txt generator writes the file that tells crawlers what they may fetch. Pick a preset, decide what AI crawlers such as GPTBot, ClaudeBot and Google-Extended may do, and download the file. The tester checks any robots.txt the way Googlebot reads it (one group per crawler, longest matching rule wins, ties go to Allow) and marks the line that decides.

The sitemap tabs turn a list of URLs into a valid sitemap.xml and check an existing sitemap for the mistakes that make search engines skip it. Connecting a domain? The DNS lookup checks its records.

How to use the robots.txt and sitemap tools

  1. 1

    Generate

    Pick a preset (business site, WordPress, online store, web app), choose what AI crawlers may do, add your own paths. The robots.txt updates as you click.
  2. 2

    Test

    Paste any robots.txt, pick a crawler and type a URL: the tester says Allowed or Blocked, marks the deciding line and flags syntax mistakes.
  3. 3

    Add a sitemap

    Paste your page URLs to get a valid sitemap.xml (split automatically past 50,000 URLs), or paste an existing sitemap to validate it.

What robots.txt does (and what it doesn’t)

robots.txt is a plain text file at the root of a site (https://www.example.com/robots.txt) that tells crawlers which URLs they may fetch. It is a request, not a lock: Google, Bing and the large AI companies follow it; scrapers may not. Each host needs its own file, so shop.example.com is not covered by the file on www.example.com.

  • It controls crawling, not indexing. A blocked URL can still appear in Google (without a description) if other pages link to it. To keep a page out of search results, let Google crawl it and add a noindex robots meta tag.
  • It is public. Anyone can read it, so never list secret URLs in it. Protect private pages with a login.
  • Don’t block CSS and JavaScript. Google renders pages like a browser; blocking /_next/, /assets/ or *.js can make pages look broken to it.

robots.txt syntax in five lines

A file is a list of groups. Each group names one or more crawlers and gives their rules:

User-agent: *
Disallow: /admin/
Allow: /admin/help/

Sitemap: https://www.example.com/sitemap.xml
  • User-agent names the crawler by its token (Googlebot, GPTBot); * means every crawler without its own group. Names are case-insensitive.
  • Disallow and Allow take a path from the root. Paths are case-sensitive and match by prefix: Disallow: /search also blocks /search-tips.
  • * matches any characters, $ marks the end of the URL: Disallow: /*.pdf$ blocks every URL ending in .pdf.
  • Sitemap takes a full URL and applies to the whole file, not to one group. You can list several.
  • An empty Disallow: blocks nothing. Disallow: / blocks everything.

How Google decides: the rules the tester follows

The tester implements Google’s published matching rules, so its answer is the one Googlebot would reach:

  1. One group per crawler. A crawler obeys the group that names it and ignores User-agent: * completely. A User-agent: Googlebot group with only Disallow: /private/ lets Googlebot crawl /admin/ even if the * group blocks it. Several groups for the same crawler are merged.
  2. Longest match wins. Among the rules that match the URL, the one with the longest path applies. With Disallow: /shop/ and Allow: /shop/sale/, /shop/sale/shoes is allowed.
  3. Ties go to Allow. If an Allow and a Disallow rule are the same length, Google uses the less restrictive one.
  4. Order doesn’t matter. Unlike some old crawlers, Google doesn’t use the first matching line.
  5. Only four fields count. Google reads user-agent, allow, disallow and sitemap, and ignores Crawl-delay, Host and Noindex. It reads the first 500 KiB of the file and caches it for up to 24 hours.

Google’s own image, news and video crawlers (Googlebot-Image, -News, -Video) obey a Googlebot group when they have none of their own; the tester applies that fallback. Bing and other engines follow the same longest-match idea, but details can differ.

AI crawlers: GPTBot, ClaudeBot, Google-Extended and the rest

AI companies use separate tokens for separate jobs, so you can opt out of model training and still be cited in AI search answers. These are the tokens each company documents (checked November 2026):

TokenCompanyPurposeWhat it does
GPTBotOpenAIModel trainingCollects content that may be used to train OpenAI’s models.
OAI-SearchBotOpenAIAI searchLets your pages appear in ChatGPT search answers.
ChatGPT-UserOpenAIUser-requested fetchOpens a page when a ChatGPT user asks. OpenAI says robots.txt rules may not apply.
ClaudeBotAnthropicModel trainingCollects public content that may be used to train Claude models.
Claude-SearchBotAnthropicAI searchCrawls to improve Claude’s search answers.
Claude-UserAnthropicUser-requested fetchFetches a page when a Claude user’s question needs it.
Google-ExtendedGoogleModel trainingNot a separate crawler: a token for Gemini training and grounding. No effect on Google Search.
Applebot-ExtendedAppleModel trainingDoesn’t crawl: opts out of Apple model training. Applebot (search) is separate.
meta-externalagentMetaModel trainingCollects content to train Meta’s AI models.
CCBotCommon CrawlModel trainingBuilds the open Common Crawl archive, widely used to train AI models.
PerplexityBotPerplexityAI searchShows and links your pages in Perplexity answers.
meta-webindexerMetaAI searchIndexes pages to cite and link them in Meta AI answers.
Perplexity-UserPerplexityUser-requested fetchFetches a page a user asked about. Perplexity says it generally ignores robots.txt.
meta-externalfetcherMetaUser-requested fetchFetches a link a user asked about. Meta says it may bypass robots.txt.
  • Blocking Google-Extended doesn’t affect Google Search. Google says it is not a ranking signal and doesn’t change whether your pages are in Search. It is not a separate crawler: Googlebot fetches the page, the token says whether Gemini may use it.
  • User-requested fetchers are different. ChatGPT-User, Perplexity-User and meta-externalfetcher act for a person who asked about your page; their operators say robots.txt may not apply.
  • robots.txt works going forward. It doesn’t remove content already collected.

XML sitemaps: format and limits

A sitemap lists the URLs you want search engines to find. It helps most on new sites, large sites and pages with few internal links. The format, from sitemaps.org:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://www.example.com/pricing</loc>
    <lastmod>2026-10-20</lastmod>
  </url>
</urlset>
  • Up to 50,000 URLs and 50 MB (uncompressed) per file; beyond that, split it and list the files in a sitemap index.
  • Full, absolute URLs on the same host as the sitemap, written exactly as your site serves them (https, with or without www).
  • Escape & as &amp;; a bare & breaks the whole file.
  • lastmod in W3C format (2026-10-20 or 2026-10-20T14:30:00+00:00). Google uses it only if it is consistently accurate, and ignores changefreq and priority.
  • List only canonical pages that return 200: no redirects, no noindex pages, no URLs blocked in robots.txt.

Tell search engines where it is with a Sitemap: line in robots.txt and by submitting it in Google Search Console and Bing Webmaster Tools.

Common robots.txt mistakes the checker catches

  • Disallow: / left over from staging. The most expensive line in SEO: the whole site stops being crawled.
  • A crawler group with no rules, which silently exempts that crawler from your * rules.
  • Rules before any User-agent line, which every crawler ignores.
  • Full URLs in Disallow (Disallow: https://example.com/private) instead of paths.
  • Noindex in robots.txt, which Google stopped reading in 2019.
  • Typos like Dissalow or User agent: Google forgives some, other crawlers don’t.
  • Relative Sitemap URLs and files over Google’s 500 KiB limit.

Where to put robots.txt on common platforms

  • Any host with file access: upload robots.txt to the web root, next to your home page’s index file.
  • WordPress: WordPress serves a virtual robots.txt; a real file in the root replaces it, and SEO plugins (Yoast, Rank Math) have a robots.txt editor.
  • Shopify: edit the robots.txt.liquid template in your theme.
  • Next.js: put the file in public/, or generate it with app/robots.ts.
  • Wix, Squarespace and other builders: look for SEO settings; some let you edit robots.txt, some don’t.

Worked examples

Block AI training, stay in AI search

Choose Block AI training. GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent and CCBot get Disallow: /; OAI-SearchBot, Claude-SearchBot and PerplexityBot have no group, so they follow your * rules and can still cite you.

Why /search-tips was blocked

Disallow: /search is a prefix, so it blocks /search-tips too. The Business preset uses /search$ (exactly /search) and /search? (search with a query) instead; test /blog/search-tips in the tester to see it stays allowed.

Open one file inside a blocked folder

With Disallow: /private/ and Allow: /private/press-kit.pdf, the press kit is allowed: the Allow path is longer (22 characters against 9), so it wins. Everything else in /private/ stays blocked.

A sitemap with 120,000 product URLs

Paste the list into the sitemap generator: it makes sitemap-1.xml to sitemap-3.xml (50,000, 50,000 and 20,000 URLs) and a sitemap-index.xml that lists them, all in one ZIP. Submit only the index in Search Console.

Frequently asked questions

How do I check my robots.txt?

Open yoursite.com/robots.txt, copy the text and paste it into the robots.txt tester tab. It flags syntax problems and, for any crawler and URL you pick, tells you whether the URL is allowed and which line decides. Google Search Console also has a robots.txt report that shows the version Google last fetched.

Is this robots.txt tester the same as Google’s?

It follows the matching rules Google publishes (one group per crawler, longest match wins, ties go to Allow, * and $ wildcards, percent-encoding), so for Googlebot it gives the same answer. Google retired its old standalone robots.txt Tester in 2023; Search Console now shows a robots.txt report instead.

Should I block GPTBot and other AI crawlers?

It is a business choice. Blocking training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot…) keeps new content out of future model training. Blocking AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) also keeps you out of their answers and citations, which can cost traffic. Many sites block training and allow search.

Does blocking Google-Extended remove my site from Google or AI Overviews?

Google says Google-Extended does not affect inclusion or ranking in Google Search. It controls whether Gemini models may use your content for training and grounding. AI features inside Google Search are governed by Googlebot, so blocking Googlebot would also remove you from Search.

Does Disallow remove a page from Google?

No. Disallow stops crawling; a blocked URL can still be indexed from links, just without a description. To remove a page, allow crawling and add a noindex meta tag (or X-Robots-Tag header), or use the Removals tool in Search Console for a quick temporary removal.

Do I need a robots.txt file?

No. Without one (a 404), crawlers assume everything may be crawled. Add one when you want to keep areas like admin, cart or internal search out of the crawl, set AI crawler rules, or point to your sitemap.

How do I generate a sitemap without crawling my site?

Export your page URLs (from your CMS, a spreadsheet or a crawler) and paste them into the sitemap generator. It removes duplicates and #fragments, skips anything that is not a full http(s) URL, escapes special characters, and splits large lists into files of 50,000 URLs with an index.

Why does my sitemap validator say the namespace is wrong?

The root element must say xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" exactly. A common mistake is https:// or a trailing slash; search engines then don’t recognise the file as a sitemap.

Updated November 2026

Related free tools

All tools →

DNS Lookup and Domain Checker

Check DNS records, email setup, domain status and propagation, and see which records to change.

Schema Markup Generator

Pick a schema type, fill in the form and copy JSON-LD checked against Google’s structured data rules.

SEO and Website Launch Checklist

A website launch and SEO checklist with every item explained, progress saved in your browser and Markdown or PDF export.

SPF, DKIM and DMARC Checker

Grade a domain’s email authentication, see each SPF, DKIM and DMARC record explained and copy the DNS records that fix it.

Favicon Generator

Make a favicon from text, an emoji or an image and download every size, plus the HTML to paste.

Ventora

Rules are set. Now build the site

Describe your business and Ventora builds the website, the landing page and the checkout.

Plain words are fine. Edit it before you start.

200 free credits, no card.

  • A working product and landing page you can publish
  • Payments, hosting and custom-domain setup built in
  • Describe it in plain words, no code

The AI-powered platform that turns ideas into revenue-generating products.

Product

  • Dashboard
  • Launch
  • Pricing
  • Cases
  • Blog
  • Made with Ventora
  • Free tools

Company

  • About
  • Investors

Legal

  • Terms of Use
  • Privacy Policy
  • Cookie Policy
  • Data Processing Agreement

For data, legal or abuse inquiries, contact: support@ventora.cc

© 2026 All rights reserved.