What is a robots.txt file?

A robots.txt file is a plain-text file placed in the root of your website (e.g., https://example.com/robots.txt) that tells search engine crawlers which pages or sections they are allowed or not allowed to access. It uses the Robots Exclusion Protocol (REP) and is the standard way to communicate crawl instructions to bots like Googlebot and Bingbot.

Does blocking a URL in robots.txt remove it from search results?

No. Disallowing a URL in robots.txt prevents crawlers from visiting it, but the URL can still appear in search results if other sites link to it. To remove a page from search results entirely, use a noindex meta tag or HTTP header on the page itself, or use Google Search Console's URL removal tool.

What is the difference between Allow and Disallow?

Disallow tells a bot not to crawl the specified path. Allow explicitly permits a path even if a broader Disallow rule would block it. For example, you can Disallow: /private/ and then Allow: /private/public-section/ to carve out an exception. Allow rules take precedence when both match a URL.

What does Crawl-delay do?

Crawl-delay tells a bot to wait a specified number of seconds between consecutive requests to your server. It is useful for sites with limited server resources. Note that Googlebot ignores Crawl-delay in robots.txt — use Google Search Console's crawl rate settings instead. Bingbot and most other bots do respect it.

Where should I place the robots.txt file?

The robots.txt file must be placed in the root directory of your website — accessible at https://yourdomain.com/robots.txt. It cannot be placed in subdirectories. Each subdomain (e.g., blog.example.com) needs its own robots.txt at its own root.

Generateur de Robots.txt

Creez un fichier robots.txt pour votre site web avec un editeur visuel. Controlez quels robots des moteurs de recherche peuvent explorer vos pages.

Presets rapides

User-agent:

Crawl-delay:secondes (optionnel)

Bloc agent #1

User-agent:

Crawl-delay:secondes (optionnel)

Bloc agent #2

User-agent:

Crawl-delay:secondes (optionnel)

Bloc agent #3

URL du sitemap (optionnel)

Robots.txt genere

# Generated by ToolBox Hub

User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

Enregistrez ce contenu sous robots.txt et placez-le a la racine de votre site web.

Comment utiliser

Creez un fichier robots.txt pour votre site web avec un editeur visuel. Controlez quels robots des moteurs de recherche peuvent explorer vos pages.

1Choose a preset (Allow All, Block All, or SEO-Friendly) to load a starting configuration instantly.
2Customize each User-agent block: change the bot name, add Allow or Disallow path rules, and set an optional Crawl-delay.
3Add more User-agent blocks with the 'Add User-agent Block' button to target specific bots with different rules.
4Optionally enter your Sitemap URL to include a Sitemap directive in the output.
5Click 'Copy' to copy the generated robots.txt to your clipboard, then save it as robots.txt in your website's root directory.

Fonctionnalités

Three quick presets: Allow All, Block All, and SEO-Friendly (Google + Bing)
Add multiple User-agent blocks, each with their own Allow/Disallow rules and Crawl-delay
Autocomplete suggestions for common bots (Googlebot, Bingbot, AhrefsBot, and more)
Optional Sitemap URL field for the Sitemap directive
Live robots.txt preview updates as you type — copy with one click

Generateur de Robots.txt

Comment utiliser

Fonctionnalités

Questions fréquentes

En savoir plus