What is a robots.txt file?

A robots.txt file is a plain-text file placed in the root of your website (e.g., https://example.com/robots.txt) that tells search engine crawlers which pages or sections they are allowed or not allowed to access. It uses the Robots Exclusion Protocol (REP) and is the standard way to communicate crawl instructions to bots like Googlebot and Bingbot.

Does blocking a URL in robots.txt remove it from search results?

No. Disallowing a URL in robots.txt prevents crawlers from visiting it, but the URL can still appear in search results if other sites link to it. To remove a page from search results entirely, use a noindex meta tag or HTTP header on the page itself, or use Google Search Console's URL removal tool.

What is the difference between Allow and Disallow?

Disallow tells a bot not to crawl the specified path. Allow explicitly permits a path even if a broader Disallow rule would block it. For example, you can Disallow: /private/ and then Allow: /private/public-section/ to carve out an exception. Allow rules take precedence when both match a URL.

What does Crawl-delay do?

Crawl-delay tells a bot to wait a specified number of seconds between consecutive requests to your server. It is useful for sites with limited server resources. Note that Googlebot ignores Crawl-delay in robots.txt — use Google Search Console's crawl rate settings instead. Bingbot and most other bots do respect it.

Where should I place the robots.txt file?

The robots.txt file must be placed in the root directory of your website — accessible at https://yourdomain.com/robots.txt. It cannot be placed in subdirectories. Each subdomain (e.g., blog.example.com) needs its own robots.txt at its own root.

Robots.txt Generator

Erstellen Sie eine robots.txt-Datei für Ihre Website mit einem visuellen Editor. Legen Sie fest, welche Suchmaschinen-Bots Ihre Seiten crawlen dürfen.

Schnell-Vorlagen

User-agent:

Crawl-delay:Sekunden (optional)

Agenten-Block #1

User-agent:

Crawl-delay:Sekunden (optional)

Agenten-Block #2

User-agent:

Crawl-delay:Sekunden (optional)

Agenten-Block #3

Sitemap-URL (optional)

Generierte robots.txt

# Generated by ToolBox Hub

User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

Speichern Sie diesen Inhalt als robots.txt und legen Sie ihn im Stammverzeichnis Ihrer Website ab.

Anleitung

Erstellen Sie eine robots.txt-Datei für Ihre Website mit einem visuellen Editor. Legen Sie fest, welche Suchmaschinen-Bots Ihre Seiten crawlen dürfen.

1Choose a preset (Allow All, Block All, or SEO-Friendly) to load a starting configuration instantly.
2Customize each User-agent block: change the bot name, add Allow or Disallow path rules, and set an optional Crawl-delay.
3Add more User-agent blocks with the 'Add User-agent Block' button to target specific bots with different rules.
4Optionally enter your Sitemap URL to include a Sitemap directive in the output.
5Click 'Copy' to copy the generated robots.txt to your clipboard, then save it as robots.txt in your website's root directory.

Funktionen

Three quick presets: Allow All, Block All, and SEO-Friendly (Google + Bing)
Add multiple User-agent blocks, each with their own Allow/Disallow rules and Crawl-delay
Autocomplete suggestions for common bots (Googlebot, Bingbot, AhrefsBot, and more)
Optional Sitemap URL field for the Sitemap directive
Live robots.txt preview updates as you type — copy with one click

Robots.txt Generator

Anleitung

Funktionen

Häufig gestellte Fragen

Mehr erfahren