What is a robots.txt file?

A robots.txt file is a plain-text file placed in the root of your website (e.g., https://example.com/robots.txt) that tells search engine crawlers which pages or sections they are allowed or not allowed to access. It uses the Robots Exclusion Protocol (REP) and is the standard way to communicate crawl instructions to bots like Googlebot and Bingbot.

Does blocking a URL in robots.txt remove it from search results?

No. Disallowing a URL in robots.txt prevents crawlers from visiting it, but the URL can still appear in search results if other sites link to it. To remove a page from search results entirely, use a noindex meta tag or HTTP header on the page itself, or use Google Search Console's URL removal tool.

What is the difference between Allow and Disallow?

Disallow tells a bot not to crawl the specified path. Allow explicitly permits a path even if a broader Disallow rule would block it. For example, you can Disallow: /private/ and then Allow: /private/public-section/ to carve out an exception. Allow rules take precedence when both match a URL.

What does Crawl-delay do?

Crawl-delay tells a bot to wait a specified number of seconds between consecutive requests to your server. It is useful for sites with limited server resources. Note that Googlebot ignores Crawl-delay in robots.txt — use Google Search Console's crawl rate settings instead. Bingbot and most other bots do respect it.

Where should I place the robots.txt file?

The robots.txt file must be placed in the root directory of your website — accessible at https://yourdomain.com/robots.txt. It cannot be placed in subdirectories. Each subdomain (e.g., blog.example.com) needs its own robots.txt at its own root.

Robots.txt 生成器

使用可视化编辑器为您的网站创建 robots.txt 文件，控制搜索引擎爬虫可以抓取的页面。

快速预设

User-agent:

Crawl-delay:秒（可选）

代理块 #1

User-agent:

Crawl-delay:秒（可选）

代理块 #2

User-agent:

Crawl-delay:秒（可选）

代理块 #3

网站地图 URL (可选)

生成的 robots.txt

# Generated by ToolBox Hub

User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

将此内容保存为 robots.txt 文件并放置在网站根目录中。

使用方法

使用可视化编辑器为您的网站创建 robots.txt 文件，控制搜索引擎爬虫可以抓取的页面。

1Choose a preset (Allow All, Block All, or SEO-Friendly) to load a starting configuration instantly.
2Customize each User-agent block: change the bot name, add Allow or Disallow path rules, and set an optional Crawl-delay.
3Add more User-agent blocks with the 'Add User-agent Block' button to target specific bots with different rules.
4Optionally enter your Sitemap URL to include a Sitemap directive in the output.
5Click 'Copy' to copy the generated robots.txt to your clipboard, then save it as robots.txt in your website's root directory.

主要功能

Three quick presets: Allow All, Block All, and SEO-Friendly (Google + Bing)
Add multiple User-agent blocks, each with their own Allow/Disallow rules and Crawl-delay
Autocomplete suggestions for common bots (Googlebot, Bingbot, AhrefsBot, and more)
Optional Sitemap URL field for the Sitemap directive
Live robots.txt preview updates as you type — copy with one click

Robots.txt 生成器

使用方法

主要功能

常见问题

了解更多