Indexability: robots.txt and meta robots

What are robots.txt and meta robots?

robots.txt is a plain text file located in the site’s root directory (e.g., https://example.com/robots.txt) that tells search engine bots which directories or pages they are allowed to crawl. The meta robots tag, on the other hand, is located in the HTML <head> section and controls whether a page will be indexed in search results even if it has been crawled. They operate on different levels: one answers the question “Can you enter through this door?”, while the other answers “Can you add this room to the catalog after entering?”

This distinction is often confused because both are discussed under the heading of “managing bots,” but their scopes are entirely different, and the wrong combination can lead to unintended consequences.

robots.txt is actually a “set of rules,” not a security measure: well-intentioned bots (such as Googlebot) follow these rules, but malicious bots can completely ignore the file. Therefore, trying to “hide” confidential or sensitive content using robots.txt is a flawed security assumption.

Why is this important?

Confusing these two mechanisms can seriously harm a site’s visibility. If a page is blocked from being crawled via robots.txt, the search engine may never see its noindex tag — because it must first crawl the page to see the tag — and the page may inadvertently remain in the index (usually as an entry without a title or description).

When used correctly, however, these two tools allow you to precisely control which content appears in search results: robots.txt keeps the crawl budget away from unimportant areas (e.g., the admin panel, search results pages), while meta robots ensures that specific pages (e.g., thank-you pages, filtered listing variations) are removed from the index.

On large sites, the crawl budget is a limited resource: the bot crawls only a portion of the site during each visit. Blocking low-value areas (e.g., internal search result pages, filter combinations) with robots.txt ensures that the bot spends its time on the pages that really matter.

How to fix it?

Only block directories you truly don’t want crawled (e.g., admin panels, shopping cart pages) using robots.txt.
If you want to remove a page from the index, add <meta name="robots" content="noindex"> to that page.
Remember that for noindex to work, the page must be allowed to be crawled by robots.txt.
Adhere to the single source of truth principle: do not repeat the same signal in conflicting ways using multiple methods.
Check your robots.txt file regularly; test or staging restrictions may have accidentally remained in the live environment.
Use the Coverage report in Google Search Console to regularly check which pages are marked as “not crawled” or “excluded due to noindex.”

Example

User-agent: *
Disallow: /admin/
Allow: /

<meta name="robots" content="noindex, follow">

Bad example: First adding Disallow: /campaign/ in robots.txt, and then also adding the noindex tag to the page itself with <meta name="robots" content="noindex"> — since the bot won’t crawl the page at all, the noindex tag won’t work, and the page may still appear in the index.

Common mistakes

Accidentally blocking the entire site from being crawled with Disallow: /.
Blocking a page with robots.txt and also adding a noindex tag (the tag is never seen, so the page may remain in the index).
Forgetting to move the restrictive robots.txt file from the test/staging environment to the live site.
Accidentally adding important pages to the "Disallow" list.
Migrating old robots.txt rules to the new site without reviewing them after a site migration or redesign; a restriction that made sense on the old site may also cover important pages in the new structure.
Failing to establish a consistent indexing policy for pages that generate pagination, filter, or sorting parameters; when some of these pages remain in the index while others are excluded, it creates mixed signals across the site.
Using the canonical tag and the noindex directive in a conflicting manner on the same page—for example, setting a page to noindex while designating another page as canonical—creates mixed signals.
Forgetting that a single robots.txt file is sufficient for the entire site and attempting to set separate access restrictions in subdirectories; robots.txt applies only to the root directory.
In the Seoraporu.co report, conflicting signals such as “blocked by robots.txt but also has noindex” are flagged separately; instead of keeping both on the same page, choose only one.