SEO & Crawler Tool
Create search-engine-compliant robots.txt files with custom crawler directives, delay rules, sitemap declarations, and modern AI scraper controls.
Select a popular template to automatically configure standard robots.txt directives:
User-agent: *)Allow all robots to crawl, except for specific disallowed folders below.
Sets the minimum seconds between crawler requests to avoid overloading your server.
Disallow:)Search engine bots will NOT crawl any URL that begins with these paths.
Allow:)Grant permission to specific files or sub-folders located inside a restricted folder (e.g. /wp-admin/admin-ajax.php).
No custom allow exceptions added.
(GPTBot)OpenAI crawler used to train and feed ChatGPT models.
(ClaudeBot)Anthropic crawler used for Claude training & data.
(CCBot)Common Crawl bot used widely across open AI model training.
(Google-Extended)Google crawler used specifically for Gemini & AI training.
(Bytespider)ByteDance scraper for TikTok and AI model data.
(PerplexityBot)Perplexity AI real-time search and answer engine crawler.
Sitemap:)Declaring your XML sitemap URL helps search engines discover all indexable pages on your site.
Place this file at the root directory of your web server (e.g. https://yourdomain.com/robots.txt).
# robots.txt generated by Free Tools User-agent: * Disallow: /admin/ Disallow: /private/ Sitemap: https://example.com/sitemap.xml
Craft clean, standardized crawler rules that optimize search engine visibility while protecting server resources.
Prevent search engine bots from wasting crawl bandwidth on duplicate query URLs, temporary files, and internal search results.
Easily control or block automated AI scrapers and LLM training bots like GPTBot, ClaudeBot, CCBot, Google-Extended, and Bytespider.
Configure custom allow and disallow directives for individual search bots including Googlebot, Bingbot, Yandex, Baidu, and DuckDuckGo.
All directives are generated instantly in your browser with zero latency, no tracking, and no signup needed.
A quick guide to deploying your robots.txt across popular hosting environments and frameworks.
Save the generated text as robots.txt and upload it to your web server's root folder (usually public_html/ or /var/www/html/).
Place the file in your WordPress root directory, or paste the rules into your SEO plugin (Yoast SEO Tools > File Editor or Rank Math > General Settings > Edit robots.txt).
Place robots.txt inside your project's public/ directory or define dynamic rules in app/robots.ts.
Key questions about crawler directives, bot behaviors, and search index optimization.
A robots.txt file is a plain text file placed in the root directory of your website. It uses the Robots Exclusion Standard to tell web search crawlers which URLs and directories they are permitted or forbidden to access.
No. Robots.txt tells well-behaved crawlers not to fetch a page, but malicious bots may ignore it. Furthermore, if an external site links to your disallowed URL, search engines might still index the URL without crawling the page content. For absolute privacy, use password protection or noindex headers.
A 'Disallow' directive in robots.txt prevents search engine spiders from crawling the page. A 'noindex' meta tag tells search engines they may crawl the page but must not show it in search results.
Crawl-delay requests that search crawlers wait a specified number of seconds between consecutive page requests to reduce server load. Note that Bing and Yandex support crawl-delay, while Google uses automatic crawl rate algorithms configured in Google Search Console.
Upload the file to the highest-level root directory of your website hosting (e.g. https://yourdomain.com/robots.txt). It will not be recognized if placed in a subfolder like /assets/robots.txt.
You can test your file using Google Search Console's Robots Testing Tool or by directly opening https://yourdomain.com/robots.txt in your web browser.