Robots.txt: control crawling without hiding your site

Robots.txt is a public text file at the root of a site. It tells compliant crawlers which paths they may request. A careful file can reduce wasted crawling, but one broad rule can also block search engines from important content.

How the rules work

Each group starts with a User-agent and contains Allow or Disallow rules. An asterisk applies to crawlers without a more specific group. An empty Disallow allows everything; Disallow followed by a single slash blocks the entire site.

• Use paths that begin with a slash, such as /admin/ or /checkout/.
• Keep rules as narrow as possible and test important addresses before publishing.
• Add the full sitemap address on a separate Sitemap line.

Robots.txt does not remove pages from search

Blocking crawling is not the same as blocking indexing. A search engine can still know about a blocked address through links and may show the URL without a useful description. Use a noindex directive on a crawlable page when the goal is removal from search results.

Do not use it for private information

Anyone can read robots.txt, and malicious bots may ignore it. Passwords, customer data, private documents, and administration areas need real access controls rather than crawler instructions.

Check every change

Create a clean file with our Robots.txt Generator, then paste or load it in the Robots.txt Tester. Test your home page, key landing pages, images, and other crawlable assets for the crawlers that matter.