What is a robots.txt file?

A robots.txt file is a plain-text file placed in the root of your website (e.g., https://example.com/robots.txt) that tells search engine crawlers which pages or sections they are allowed or not allowed to access. It uses the Robots Exclusion Protocol (REP) and is the standard way to communicate crawl instructions to bots like Googlebot and Bingbot.

Does blocking a URL in robots.txt remove it from search results?

No. Disallowing a URL in robots.txt prevents crawlers from visiting it, but the URL can still appear in search results if other sites link to it. To remove a page from search results entirely, use a noindex meta tag or HTTP header on the page itself, or use Google Search Console's URL removal tool.

What is the difference between Allow and Disallow?

Disallow tells a bot not to crawl the specified path. Allow explicitly permits a path even if a broader Disallow rule would block it. For example, you can Disallow: /private/ and then Allow: /private/public-section/ to carve out an exception. Allow rules take precedence when both match a URL.

What does Crawl-delay do?

Crawl-delay tells a bot to wait a specified number of seconds between consecutive requests to your server. It is useful for sites with limited server resources. Note that Googlebot ignores Crawl-delay in robots.txt — use Google Search Console's crawl rate settings instead. Bingbot and most other bots do respect it.

Where should I place the robots.txt file?

The robots.txt file must be placed in the root directory of your website — accessible at https://yourdomain.com/robots.txt. It cannot be placed in subdirectories. Each subdomain (e.g., blog.example.com) needs its own robots.txt at its own root.

Robots.txt Generator

Create a robots.txt file for your website with a visual editor. Control which search engine bots can crawl your pages.

Quick Presets

User-agent:

Crawl-delay:seconds (optional)

Agent block #1

User-agent:

Crawl-delay:seconds (optional)

Agent block #2

User-agent:

Crawl-delay:seconds (optional)

Agent block #3

Sitemap URL (optional)

Generated robots.txt

# Generated by ToolBox Hub

User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

Save this content as robots.txt and place it in the root of your website.

How to Use

Create a robots.txt file for your website with a visual editor. Control which search engine bots can crawl your pages.

1Choose a preset (Allow All, Block All, or SEO-Friendly) to load a starting configuration instantly.
2Customize each User-agent block: change the bot name, add Allow or Disallow path rules, and set an optional Crawl-delay.
3Add more User-agent blocks with the 'Add User-agent Block' button to target specific bots with different rules.
4Optionally enter your Sitemap URL to include a Sitemap directive in the output.
5Click 'Copy' to copy the generated robots.txt to your clipboard, then save it as robots.txt in your website's root directory.

Key Features

Three quick presets: Allow All, Block All, and SEO-Friendly (Google + Bing)
Add multiple User-agent blocks, each with their own Allow/Disallow rules and Crawl-delay
Autocomplete suggestions for common bots (Googlebot, Bingbot, AhrefsBot, and more)
Optional Sitemap URL field for the Sitemap directive
Live robots.txt preview updates as you type — copy with one click

Robots.txt Generator

How to Use

Key Features

Frequently Asked Questions

Learn More