Tip 3 of 31: Check That Your Pages Are Not Hidden

You could have the best content on your website and still be absent from search results. A single hidden instruction can be enough to prevent an important page from being indexed.

The Problem

A surprisingly common reason a page does not appear in search results is a small instruction telling supported search engines not to include it in their index. That instruction is called noindex.

It is not always placed deliberately. Pages may be marked noindex during development to prevent staging environments from being indexed, or through plugins and CMS settings that apply rules across multiple pages. Once the website goes live, those instructions sometimes remain in place.

What Noindex Actually Does

Imagine writing an important article and then attaching a note that says: “Please do not put this in the library catalogue.”

That is effectively what a noindex instruction does. When a supported search engine crawls the page and detects the instruction, it will not include that page in its search index.

A noindex instruction can appear in a robots meta tag within the page HTML or in an X-Robots-Tag HTTP response header.

The page must normally remain accessible to the crawler so it can detect the instruction. Blocking the same URL through robots.txt can prevent the crawler from seeing the noindex rule.

Noindex has legitimate uses, including private pages, thank-you pages after form submissions, internal administration content and some utility pages. It becomes a serious problem when applied to a service page, case study, guide or other content you want potential customers and candidates to find.

Why This Matters

We have seen accidental noindex instructions cause real visibility losses.

One of our clients generated £45,000 in placement fees from their website in a single month. Results like this depend on important pages being available for people and search engines to find.

A page excluded from a search engine’s index cannot appear in results from that index. This may also reduce its availability to systems that rely on search indexes, although different AI-assisted systems use different crawlers, indexes and sources.

Removing an accidental noindex instruction makes the page eligible to be considered for indexing. It does not guarantee that the page will be indexed, rank prominently or receive citations.

What to Do

Ask your developer or SEO specialist to audit the noindex status of your important pages.

Google Search Console’s Page indexing report can identify pages excluded because of a noindex instruction. For an individual page, use the URL Inspection tool and check whether indexing is allowed.

Look particularly at:

• Service pages
• Blog posts and guides
• Landing pages
• Case studies
• Sector and location pages
• Recently migrated or redesigned content

Your developer should check:

• Robots meta tags in the page HTML
• X-Robots-Tag HTTP response headers
• CMS visibility settings
• SEO plugin settings
• Template-level instructions
• Differences between staging and live environments

Any page you want a potential client or candidate to find through search should normally be eligible for indexing.

Do not remove noindex from every page automatically. Thank-you pages, private content, internal search results and administrative pages may be intentionally excluded.

After correcting an accidental instruction, use the URL Inspection tool to test the live page and request indexing where appropriate. The search engine will still need to recrawl and reassess the page.

Up Next

Tip 4 of 31 covers a related problem: pages that can be found but have been instructed not to show a description or extract in supported search results.

Frequently Asked Questions

What is a noindex tag?

A noindex instruction tells supported search engines not to include a page in their search index. It can be placed in a robots meta tag within the page HTML or supplied through an X-Robots-Tag HTTP response header. The page must normally remain crawlable so the search engine can detect the instruction.

How would a noindex tag end up on my pages by mistake?

Common causes include development settings that were not removed after launch, SEO plugins applying broad rules, CMS settings affecting whole groups of pages, template-level instructions and staging settings carried over to the live website. Check again after platform migrations, redesigns and major updates.

How do I find out if any of my pages are noindexed?

Google Search Console’s Page indexing report can show pages excluded by a noindex instruction. For an individual URL, use the URL Inspection tool and check whether indexing is allowed. A developer can also crawl the live website and inspect its HTML and HTTP response headers.

Should some pages on my website be noindexed?

Yes. Thank-you pages, private or account-specific content, internal search results and some administrative or utility pages may be suitable. The decision should be deliberate, and the instruction should not be applied to service pages, useful articles, case studies or other content you want people to find through search.

Does noindex affect AI citations as well as search rankings?

A page excluded from a search engine’s index cannot be surfaced from that index, which may also reduce its availability to systems that rely on those search results. However, AI-assisted systems use different crawlers, indexes and sources, so noindex should not be treated as a universal control over every AI citation. Removing an accidental noindex instruction makes a page eligible for indexing but does not guarantee rankings or citations.

Darren Revell, Co-Founder, RecruiterWEB