Robots.txt: How to Use It Correctly

Robots.txt is a plain text file at the root of your site that tells search engine crawlers which URLs they may and may not request. It controls crawling, not indexing, and one wrong line can quietly hide your entire website from Google.

The robots txt file is one of the smallest files on your website and one of the most dangerous to get wrong. Google is blunt about its purpose: “A robots.txt file tells search engine crawlers which URLs the crawler can access on your site.” That is it. It is a traffic instruction for bots, not a lock on your content, and confusing the two is where costly mistakes begin.

Key Takeaways

• Robots.txt controls crawling, not indexing. It tells crawlers what to fetch. It does not reliably remove a page from Google’s search results.
• Blocked pages can still be indexed. If other sites link to a disallowed URL, Google can still list it, often with no description.
• To keep a page out of Google, use noindex, not Disallow. The two tools solve different problems.
• One line can break everything. A stray Disallow: / tells every crawler to ignore your whole site.
• It works with your sitemap. Reference your sitemap inside robots.txt so crawlers find your important URLs faster.

Table of Contents

• What Is Robots.txt?
• The Correct Robots.txt Syntax
• Blocking Crawling Is Not the Same as Noindex
• Example: A Correct Robots.txt File
• The Costly Mistakes to Avoid
• How Robots.txt Works With Your Sitemap
• The Robots.txt Checklist
• Frequently Asked Questions

What Is Robots.txt?

Robots.txt is a single UTF-8 text file that lives at the root of your domain, always at yoursite.com/robots.txt. When a crawler such as Googlebot visits your site, it reads this file first to learn which paths it is allowed to request. According to Google’s documentation, the file must sit at the root of the host, there can be only one per site, and rules are case-sensitive.

Think of it as a polite sign at the front door, not a lock. Reputable crawlers like Googlebot obey it. It manages crawl traffic so bots do not overload your server or waste time on low-value URLs like admin pages, internal search results or faceted filters. What it does not do is guarantee secrecy or removal from search. That distinction is the single most important thing to understand about the robots txt file, and we come back to it below.

The Correct Robots.txt Syntax

A robots.txt file is built from a handful of directives, grouped by which crawler they apply to. Google processes these groups from top to bottom and applies the first, most specific group that matches a given crawler. Here are the four directives that matter.

Directive	What it does
User-agent	Names the crawler the following rules apply to. An asterisk (*) matches all crawlers.
Disallow	Tells the named crawler not to request URLs starting with that path.
Allow	Overrides a Disallow to permit a specific path inside a blocked directory.
Sitemap	Points crawlers to your XML sitemap. It must be a full, absolute URL.

A few rules keep you out of trouble, and they are set out in full in Google’s robots.txt specification. The file must be named exactly robots.txt and be plain text. Paths are case-sensitive, so /Admin/ and /admin/ are treated as different. Lines starting with # are comments and are ignored. Google also enforces a file size limit of 500 kibibytes, roughly 0.5 MB, and anything beyond that is ignored, though very few sites ever come close to that ceiling.

Blocking Crawling Is Not the Same as Noindex

This is the mistake that quietly damages the most websites. People assume that adding a page to robots.txt hides it from Google. It does not. Google states plainly that robots.txt “is not a mechanism for keeping a web page out of Google.”

Here is why. A Disallow rule stops Googlebot from crawling the page content. But if another website links to that URL, Google can still index the address and show it in search results, usually as a bare link with no description because the crawler was never allowed to read the page. You end up with the worst of both worlds: the page still appears in search, but it looks broken and uninformative.

The rule of thumb:

• Want a page crawled but not overloading your server, or simply off-limits to bots? Use Disallow in robots.txt.
• Want a page kept out of Google’s index entirely? Use a noindex meta tag or HTTP header on the page, as set out in Google’s guidance on blocking indexing, and crucially, do not also block it in robots.txt.

That last point catches people out. If you block a page in robots.txt, Googlebot can never crawl it, which means it can never see your noindex tag, which means the page may stay indexed. To deindex a page, you have to let Google crawl it so it can read the instruction to drop it. If your pages are stuck in the index or missing from it, our guide on how to fix Google indexing issues walks through the diagnosis step by step.

Example: A Correct Robots.txt File

Here is a clean, correct robots.txt for a typical WordPress business site. It blocks the admin area from crawlers, makes one sensible exception, and points to the sitemap:

User-agent: *
Disallow: /wp-admin/
Disallow: /cart/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com.my/sitemap_index.xml

Read line by line, this is what each instruction does:

• User-agent: * applies the rules below to every crawler. The asterisk is a wildcard for all bots.
• Disallow: /wp-admin/ stops crawlers wasting time on your WordPress dashboard, which has no business in search results.
• Disallow: /cart/ keeps the shopping cart and other transactional, no-value pages out of crawl paths.
• Allow: /wp-admin/admin-ajax.php makes one exception inside the blocked folder. That file is needed for parts of the site to load correctly, so you explicitly let crawlers reach it.
• Sitemap: https://example.com.my/sitemap_index.xml hands crawlers a direct map of your important URLs, using a full absolute address.

Now the one line that can wreck a site. This single directive tells every crawler to ignore your entire website:

User-agent: *
Disallow: /

That trailing slash means “the whole site.” It is catastrophic because it blocks Google from crawling every page you own. Over time your pages drop out of search, organic traffic collapses, and enquiries dry up. This exact line is often left over from a staging or development site, where blocking crawlers is correct, and then accidentally pushed live during a launch. If you take one thing from this guide, it is this: after any new website goes live, check that your production robots.txt does not contain Disallow: /.

The Costly Robots.txt Mistakes to Avoid

• Blocking your whole site with Disallow: /. Usually a leftover from a staging environment. It deindexes everything over time.
• Using robots.txt to hide a page from Google. Blocked pages can still be indexed if linked elsewhere. Use noindex instead.
• Blocking a page you also want deindexed. Googlebot can never see the noindex tag on a page it is not allowed to crawl.
• Blocking CSS and JavaScript. Google needs these to render and understand your pages. Blocking them can hurt how your site is evaluated.
• Assuming it protects private data. Robots.txt is public. Listing a “secret” folder in it just advertises where it is. Use real authentication for anything sensitive.
• Wrong file location or name. It must be at the root and named exactly robots.txt, or crawlers will never find it.

How Robots.txt Works With Your Sitemap

Robots.txt and your XML sitemap are two halves of the same job: helping search engines crawl your site efficiently. Robots.txt says what not to crawl. The sitemap says which URLs you most want found. Adding a Sitemap: line to robots.txt is a simple, reliable way to make sure crawlers discover that list, in addition to submitting it in Google Search Console.

The two files should agree with each other. Never list a URL in your sitemap that you also block in robots.txt, since that sends crawlers a contradictory signal. Keep the sitemap limited to clean, indexable pages, remembering that a single sitemap can hold up to 50,000 URLs and must stay under 50MB, and keep robots.txt limited to genuinely low-value paths. For the full picture on building and submitting the sitemap side, see our complete guide to XML sitemaps. You can also read Google’s own sitemaps overview for how it treats the file.

The Robots.txt Checklist

• Your file is at yoursite.com/robots.txt and named exactly robots.txt.
• It does not contain Disallow: / on your live site.
• It does not block CSS, JavaScript or pages you want ranked.
• Pages you want deindexed use noindex, and are not also blocked here.
• It references your sitemap with a full absolute URL.
• Nothing in it contradicts what is listed in your sitemap.

Frequently Asked Questions

What is a robots txt file used for?

A robots.txt file tells search engine crawlers which URLs on your site they may and may not request. It manages crawl traffic so bots do not waste time on low-value pages like admin areas. It controls crawling, not whether a page appears in search results.

Does robots.txt remove a page from Google?

No. Google states that robots.txt is not a mechanism for keeping a page out of its index. A blocked page can still be indexed if other sites link to it, often shown as a bare link with no description. To keep a page out of Google, use a noindex tag instead.

Where should the robots.txt file be located?

At the root of your domain, always at yoursite.com/robots.txt. It cannot sit in a subfolder, there can only be one per site, and it must be named exactly robots.txt in plain text. If it is anywhere else, crawlers will not find or apply it.

What is the difference between Disallow and noindex?

Disallow, in robots.txt, stops crawlers from fetching a page but does not guarantee it stays out of search. Noindex, a meta tag or HTTP header on the page itself, tells Google to drop it from the index. For deindexing, use noindex and do not block the page in robots.txt, or Google can never read the instruction.

Can a robots.txt mistake hurt my SEO?

Yes, badly. A single Disallow: / line blocks your entire site from being crawled, which slowly removes your pages from Google and collapses organic traffic. This often slips through when a staging site’s settings go live. Always check your production robots.txt after a new site launch.

Do I need a robots.txt file at all?

Most sites benefit from one, but a small site with nothing to block can run fine without it, in which case crawlers assume they may access everything. The value comes from steering bots away from low-value paths and from pointing them to your sitemap. If you have one, the priority is making sure it blocks nothing important.

The Bottom Line

Robots.txt is a small file with outsized consequences. Use it to guide crawlers away from low-value paths and toward your sitemap, never as a way to hide pages from search. Remember that blocking crawling is not the same as noindex, keep the file at your root, and above all make sure a live site never carries Disallow: /.

Get a Free Website Health Check from Newnormz

Understanding robots.txt is one thing; knowing whether yours is quietly blocking important pages is another. Our team will audit your crawl directives, indexing and technical health, show you exactly what is costing you rankings and enquiries, and hand you a clear, prioritised fix list. No obligation, no jargon.

Request your free SEO audit and website health check, or explore our web design and development services.

Related reading: XML Sitemaps: A Complete Guide | How to Fix Google Indexing Issues