What is a robots.txt file?

A robots.txt file is a plain-text file placed in the root of your website (e.g., https://example.com/robots.txt) that tells search engine crawlers which pages or sections they are allowed or not allowed to access. It uses the Robots Exclusion Protocol (REP) and is the standard way to communicate crawl instructions to bots like Googlebot and Bingbot.

Does blocking a URL in robots.txt remove it from search results?

No. Disallowing a URL in robots.txt prevents crawlers from visiting it, but the URL can still appear in search results if other sites link to it. To remove a page from search results entirely, use a noindex meta tag or HTTP header on the page itself, or use Google Search Console's URL removal tool.

What is the difference between Allow and Disallow?

Disallow tells a bot not to crawl the specified path. Allow explicitly permits a path even if a broader Disallow rule would block it. For example, you can Disallow: /private/ and then Allow: /private/public-section/ to carve out an exception. Allow rules take precedence when both match a URL.

What does Crawl-delay do?

Crawl-delay tells a bot to wait a specified number of seconds between consecutive requests to your server. It is useful for sites with limited server resources. Note that Googlebot ignores Crawl-delay in robots.txt — use Google Search Console's crawl rate settings instead. Bingbot and most other bots do respect it.

Where should I place the robots.txt file?

The robots.txt file must be placed in the root directory of your website — accessible at https://yourdomain.com/robots.txt. It cannot be placed in subdirectories. Each subdomain (e.g., blog.example.com) needs its own robots.txt at its own root.

Robots.txt ジェネレーター

ビジュアルエディターでウェブサイト用の robots.txt ファイルを作成できます。検索エンジンボットがクロールできるページを制御しましょう。

クイックプリセット

User-agent:

Crawl-delay:秒（任意）

エージェントブロック #1

User-agent:

Crawl-delay:秒（任意）

エージェントブロック #2

User-agent:

Crawl-delay:秒（任意）

エージェントブロック #3

サイトマップ URL (任意)

生成された robots.txt

# Generated by ToolBox Hub

User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

このコンテンツを robots.txt として保存し、ウェブサイトのルートに配置してください。

使い方

ビジュアルエディターでウェブサイト用の robots.txt ファイルを作成できます。検索エンジンボットがクロールできるページを制御しましょう。

1Choose a preset (Allow All, Block All, or SEO-Friendly) to load a starting configuration instantly.
2Customize each User-agent block: change the bot name, add Allow or Disallow path rules, and set an optional Crawl-delay.
3Add more User-agent blocks with the 'Add User-agent Block' button to target specific bots with different rules.
4Optionally enter your Sitemap URL to include a Sitemap directive in the output.
5Click 'Copy' to copy the generated robots.txt to your clipboard, then save it as robots.txt in your website's root directory.

主な機能

Three quick presets: Allow All, Block All, and SEO-Friendly (Google + Bing)
Add multiple User-agent blocks, each with their own Allow/Disallow rules and Crawl-delay
Autocomplete suggestions for common bots (Googlebot, Bingbot, AhrefsBot, and more)
Optional Sitemap URL field for the Sitemap directive
Live robots.txt preview updates as you type — copy with one click

Robots.txt ジェネレーター

使い方

主な機能

よくある質問

詳しく読む