SEO Is Not Dead: How AI Finds Recruitment Content

You might have heard this claim:

If Google or Bing has not indexed your page, AI cannot cite it.

That sounds convincing. But... it is not quite right.

The statement is partly true for AI features that depend on a specific search index. For example, Google says a page must be indexed and eligible to appear in Google Search with a snippet before it can be shown as a supporting link in AI Overviews or AI Mode.

But that does not mean every AI platform depends entirely on Google.

AI systems can find, retrieve and use online information through search indexes, their own crawlers, external links, feeds, APIs, public platforms and user-supplied sources.

So the more accurate statement is:

AI cannot reliably retrieve, use or cite content it cannot discover or access.

And that is the bit that matters.

SEO has not become obsolete

SEO has always been about helping search engines discover, crawl, understand and evaluate your content. AI has not made those requirements disappear.

It has created more systems, crawlers and retrieval methods for website owners to consider.

Google says the same foundational SEO practices used for conventional search also apply to its AI features. Microsoft says Bing and Copilot rely on the same core crawling, indexing and ranking foundations as traditional search.

OpenAI, Anthropic and Perplexity also operate search crawlers or user-triggered retrieval systems capable of accessing public web pages. That does not mean every platform works in the same way. It also does not mean that appearing in a search index, feed, or social post guarantees an AI citation.

It means your content has more potential discovery routes than Google alone.

50 ways AI could discover or retrieve your content

Depending on the AI platform, its settings and the user’s request, your content could potentially be discovered, retrieved or accessed through any of the following routes:

1. Google Search index

2. Bing Search index

3. An AI provider’s own web index

4. An AI provider’s general web crawler

5. Real-time web search

6. Hyperlinks followed from pages that have already been discovered

7. XML sitemaps

8. RSS feeds

9. Atom feeds

10. IndexNow submissions

11. Search-engine URL submission tools

12. Dedicated AI search crawlers, such as OAI-SearchBot, Claude-SearchBot and PerplexityBot

13. Browser or search-provider APIs

14. Third-party search APIs

15. Common Crawl

16. Other public web archives and crawl datasets

17. Links from major news websites

18. Links from other blogs

19. Links from industry publications

20. Links from authoritative reference websites

21. Wikipedia articles and references

22. Public Reddit posts and links

23. Public LinkedIn posts containing the URL

24. Public X or Twitter posts containing the URL

25. Public Facebook posts and pages

26. YouTube video descriptions

27. Podcast show notes

28. Public Substack and newsletter archives

29. Medium articles

30. Public GitHub repositories

31. Public GitHub issues and discussions

32. Stack Overflow and Stack Exchange

33. Quora

34. Public forums

35. Public community websites

36. Press-release distribution websites

37. News syndication services

38. Content syndication partners

39. Industry directories

40. Business directories

41. Academic and specialist search databases

42. Public datasets

43. Public APIs

44. Structured data feeds

45. Product and catalogue feeds

46. Knowledge graphs

47. Entity databases

48. A user supplying the URL directly to an AI system

49. A user uploading or copying the page content into an AI system

50. A connected or private data source that the user has authorised the AI to search.

These routes are not all equal

This list does not mean that every AI platform uses all 50 routes. It does not mean that publishing a link on LinkedIn guarantees that ChatGPT will find it. It does not mean that adding a page to an XML sitemap guarantees that Google will index it. It does not mean that submitting a URL through IndexNow guarantees that Bing or Copilot will cite it.

Discovery, crawling, indexing, retrieval and citation are different stages.

• A system might discover a URL but choose not to crawl it.
• It might crawl the page but choose not to index it.
• It might index the page but decide it is not relevant or reliable enough to retrieve for a particular question.
• It might use the information to help construct an answer but not display the page as a visible citation.
• It might also display the page for one question, location or user while ignoring it for another.
• The 50 routes demonstrate that AI discovery is broader than conventional Google indexing. They do not provide 50 guaranteed shortcuts to a citation.
  
  

Search crawlers and training crawlers are not the same

This distinction is often missed. OpenAI uses OAI-SearchBot to help surface websites in ChatGPT search results. GPTBot has a different purpose and is associated with content that may be used to improve OpenAI’s generative AI models. Anthropic similarly distinguishes between Claude-SearchBot, Claude-User and ClaudeBot.

Perplexity distinguishes between PerplexityBot, which supports its search results, and Perplexity-User, which may retrieve a page following a user request. Allowing or blocking a training crawler is therefore not necessarily the same as allowing or blocking an AI search crawler.

Content being used during model training does not guarantee that the resulting AI system can identify, retrieve or cite the original page. If your objective is visibility in current AI-generated answers, live search discovery and retrieval access are usually more directly relevant than possible inclusion in a future training dataset.

Where SEO fits into this

Not every item in the list is technically SEO. Publishing a podcast, contributing to an industry publication or sharing research through a public dataset forms part of a broader digital marketing and distribution strategy. However, SEO remains one of the foundations that helps make your own website accessible and understandable when an AI search system reaches it.

Good technical and on-page SEO can help ensure that:

• Important pages return a successful status and can be accessed reliably.
• Search engines and relevant search crawlers are not blocked accidentally.
• Pages are not hidden by an unintended noindex instruction.
• Each page has a clear and consistent canonical URL.
• Important pages can be accessed via standard internal links.
• XML sitemaps contain current canonical URLs.
• Headings and page structure communicate the subject clearly.
• Important information is available as readable text.
• Structured data accurately reflects the visible page content.
• Duplicate or near-duplicate pages do not divide discovery signals.
• Old URLs redirect correctly when content moves.
• Facts, prices, services and business information remain accurate.
• The content directly answers the question implied by its title.

None of this guarantees a ranking or AI citation. It does improve the chances that eligible systems can discover, access and interpret the page correctly.

What this means for recruitment agencies

Recruitment websites contain information that candidates, clients and AI systems may all need to understand.

This can include:

• Salary guides
• Recruitment market reports
• Sector specialisms
• Location expertise
• Candidate advice
• Employer guidance
• Recruiter profile pages
• Recruitment case studies
• ATS integration information
• Job board information
• Recruitment technology comparisons
• Public job adverts
• Client and candidate FAQs

Publishing this information is only the first step.

• A useful salary guide hidden inside an inaccessible PDF may be harder to retrieve than an equivalent guide published as a clear webpage.
• A recruiter profile with no sector, location or experience information gives people and machines very little to work with.
• An ATS integration page that contains only a logo does not explain what the integration does, which users it supports or what information moves between the two platforms.
• A generic service page that simply says, “We provide tailored recruitment solutions,” does not clearly establish what the agency recruits for, where it operates or why anyone should trust it.

Your pages need to contain explicit, useful and verifiable information.

Your website is not the only place that matters

AI systems may encounter information about your recruitment business on your website and elsewhere. That could include:

• A client’s website
• A recruitment industry publication
• A business directory
• An awards website
• A podcast description
• A LinkedIn company page
• A public review platform
• A conference speaker profile
• A professional association
• A software partner’s integration directory

This makes factual consistency important. If your website says you recruit across the UK, LinkedIn says you only cover London and an old directory lists services you no longer provide, a person or AI system must decide which version to trust.

Use the same business name, specialisms, locations, service descriptions and core facts wherever practical. Do not manufacture consistency by copying the same promotional paragraph everywhere. Make the underlying facts consistent while adapting the presentation for each platform and audience.

What about structured data?

Structured data can help a search system understand what a page contains. For example, it can identify an organisation, person, article, job advert, breadcrumb trail or other supported entity. But structured data is not usually a standalone discovery channel.

A crawler must normally access the page before it can read the markup. The structured data must also match the information people can see on the page. Adding schema does not rescue inaccessible, inaccurate or unhelpful content. It does not guarantee indexing, rankings or citations.

Think of it as a label that helps explain the contents of the box. It is not a delivery service that transports the box to every AI platform.

What should recruitment businesses do?

Start with the pages that genuinely help candidates and clients make decisions. For each important page:

1. Give it a stable, descriptive URL.

2. Use a clear page title and main heading.

3. Answer the primary question near the beginning.

4. State important facts directly rather than expecting the reader to infer them.

5. Make the essential information available as readable text.

6. Link to it from relevant pages on your website.

7. Add it to your XML sitemap.

8. Check that search engines and relevant AI search crawlers can access it.

9. Keep the information accurate and current.

10. Measure whether your brand and content appear for the questions that matter to your market.

Do not publish hundreds of thin pages in the hope that something gets picked up. Create pages that answer real questions with genuine knowledge, evidence and experience. Thin content is still thin content. While AI is in its infancy, it can and does cite thin content pages, and we have seen it do that, but as it evolves, they will be dropped in favour of better content.

SEO is not dead

SEO is not dead because search engines still exist.

• It is not dead because Google’s AI features still depend on Google’s crawling and indexing systems.
• It is not dead because Bing and Copilot share a crawling and indexing foundation.
• It is not dead because AI search platforms need ways to discover, retrieve and understand current web content.
• But SEO is no longer the whole story.

Your website now sits inside a much broader information environment that includes search engines, AI crawlers, third-party publications, public platforms, datasets, APIs and user-authorised sources. The objective is not to find a magic trick that forces an AI platform to cite you. The objective is to make your knowledge:

• Discoverable
• Accessible
• Understandable
• Accurate
• Useful
• Verifiable
• Worth surfacing

SEO still plays a major part in achieving that. AI has not killed SEO. It has made good SEO part of a much bigger job.

Frequently asked questions

Does a page have to be indexed by Google before AI can cite it?

Not in every AI platform. Google says a page must be indexed and eligible to appear in Search with a snippet before it can be shown as a supporting link in Google AI Overviews or AI Mode. Other AI systems can use their own search indexes, crawlers or retrieval tools, so Google indexing is not a universal requirement.

However, making a page crawlable and indexable remains an important part of recruitment SEO.

How do ChatGPT, Claude and Perplexity discover website content?

These platforms use different combinations of search crawlers, indexes and user-triggered retrieval systems. OpenAI documents OAI-SearchBot and ChatGPT-User, Anthropic documents Claude-SearchBot and Claude-User, and Perplexity documents PerplexityBot and Perplexity-User.

The exact route used depends on the platform, the question, and whether the user has asked the system to access a specific page.

Do XML sitemaps or IndexNow guarantee AI citations?

No. An XML sitemap helps eligible crawlers discover your canonical URLs. IndexNow can notify participating systems when a URL is added, changed or removed. Neither guarantees that a page will be crawled, indexed, retrieved or cited.

Should recruitment websites allow AI search crawlers?

That depends on the website’s objectives and policies. If you want public pages to be eligible for discovery through AI search products, blocking their search crawlers may reduce that opportunity. Search crawlers, training crawlers and user-triggered retrieval agents can have different purposes. Review each provider’s current documentation before deciding which agents to allow or block.

Are SEO, AEO and GEO the same thing?

They overlap, but they are not identical. SEO helps search engines discover, understand and evaluate website content. AEO focuses on making information suitable for direct answers. GEO focuses on how content and brands appear within generative AI responses.

They share many foundations, including accessible pages, clear information, strong internal linking, factual accuracy and useful content. You should measure AI visibility rather than guess whether those efforts are working.

Official sources

• Google: AI features and your website
• Microsoft: Bing Webmaster Guidelines
• OpenAI crawler documentation
• Anthropic crawler documentation
• Perplexity crawler documentation