How do you optimize content for LLMs?

To optimize content for LLMs, do what you would do for search, then check three things the AI products add. Make sure the search crawler behind each product can fetch your pages, answer the question near the top of the page in text that sits in the HTML, and say something other pages don't, under a named author and with real dates. Google states that optimizing for generative AI search is still SEO. Nobody outside these companies knows exactly how an answer picks its sources, so this guide sticks to what each one says in its own documentation.

Is optimizing for LLMs different from SEO?#

Mostly not, according to the companies that run the search products. Google's guide says that optimizing for generative AI search is optimizing for the search experience. Its list of things to do is familiar: create content that is not a copy of what everyone else says, keep a clear technical structure, and keep local business and shop details accurate.

The same guide has a "what you don't need to do" list. It says you don't need new machine readable files, AI text files, special markup or Markdown to appear in Google Search, no requirement to break content into tiny pieces, no need to write in a specific way just for generative AI search, and that seeking inauthentic mentions across the web is not as helpful as it might seem.

That does not mean nothing is new. ChatGPT, Claude and Perplexity each use their own crawlers, with their own robots.txt names. If a crawler is blocked by mistake, the product has nothing to show. That part is a technical check, and it is the first thing to do.

Why does being crawled and indexed come first?#

Each company below describes a search crawler that fetches pages for its product. Google says its AI features need a page that is indexed and eligible to show in Google Search with a snippet. It also says both features may use a technique called query fan-out, which issues several related searches across subtopics and data sources, so one question can pull in pages for several narrower searches.

OpenAI says OAI-SearchBot is used to surface websites in ChatGPT's search features, and that sites opted out of it will not be shown in ChatGPT search answers. Perplexity says PerplexityBot is designed to surface and link websites in Perplexity's search results. Anthropic says that blocking Claude-SearchBot stops content being indexed for search and may reduce visibility and accuracy in search responses.

So the first job is the one in our technical SEO checklist: robots.txt blocks nothing important, pages return a normal 200 status, and the pages you want shown are indexed. If you have never checked, our free SEO audit covers these basics, and RankGuru checks a single page's on-page basics for free. Our study of Kolkata small business websites shows how often even those basics are missing.

Which AI crawlers are there, and what does each one do?#

Each company publishes more than one crawler, and the purposes differ. This table lists what each says in its own documentation.

Sources: Google's crawler documentation, OpenAI's bots page, Anthropic's crawler help article and Perplexity's bots guide. Crawlers and names change, so check those pages before you edit your file.

What should robots.txt say about AI crawlers?#

That is a business decision, and the documents give you the pieces for it. OpenAI says each of its settings is independent of the others, so a site can allow OAI-SearchBot and disallow GPTBot. It also says a change can take about 24 hours to take effect in its search systems. Disallowing GPTBot, OpenAI says, indicates that a site's content should not be used in training its generative AI foundation models. Anthropic describes the same split: blocking ClaudeBot signals that future content should be left out of training datasets.

The practical choice is between two positions:

• Allow the search crawlers. Allow OAI-SearchBot, Claude-SearchBot and PerplexityBot so your pages can be found and linked in those products. Blocking them has the effects the companies describe above.
• Decide on the training crawlers separately. Disallow GPTBot and ClaudeBot if you don't want your content used for training. Each company says this affects training, not its search crawler.

Google-Extended works a little differently. Google says it is a standalone token with no separate user agent, and that it does not affect a site's inclusion in Google Search or work as a ranking signal. Google's AI features page points to Google-Extended for limiting AI training and grounding in other Google systems, and to snippet controls (below) for AI Overviews and AI Mode.

Two limits matter. Robots.txt only controls crawling. Google's robots.txt guide notes that Google can't index the content of a disallowed page but may still index its URL, so use noindex on a page Google can crawl to keep it out of results. And the user-triggered visits, ChatGPT-User and Perplexity-User, are treated differently: OpenAI and Perplexity both say robots.txt rules may not apply to them.

What kind of content do these products use?#

No company publishes a recipe for being picked. What exists is Google's guidance, and it is mostly the people-first advice that predates AI answers. Google's helpful content guidance asks whether a page provides original information, reporting, research or analysis, and says that crediting other sources does not replace the need for original effort. Google's AI guide adds a plainer version: create the content yourself, based on what you know, and don't recycle what others have said.

For a small business, that means writing what only you can write. Your prices and how you arrive at them, the steps in your own process, your local rules, results you measured on your own work. A page that restates the top ten search results adds nothing to an answer that already reads them. Our content creation work starts from this question: what does this business know that a competitor's page doesn't say?

Answer the question early#

Neither Google nor the other companies we cite give a rule for page layout. We write the answer in the first paragraph because it serves a reader who wants it, and because any system that quotes a passage needs that passage to make sense alone. Use clear headings that say what the section answers. Keep one topic to a page. This is ordinary good writing, and Google says you don't need to write differently for generative AI search.

Show who wrote it and when#

Google's helpful content page says it is helpful to make it obvious to visitors who wrote a page, and it encourages accurate bylines. It also warns against changing the date of a page to look fresh when the content has not substantially changed, and against inventing author profiles. Put a real author, a published date and an updated date on guides, and change the updated date only when the page changed.

Does structured data help with AI answers?#

Google says it isn't required for generative AI search, and that there is no special schema.org markup to add. What it does ask is that structured data matches the visible text on the page. So mark up what is really there, such as your organization, an article's author and date, or product details, and don't add fields the page doesn't show. The markup still serves its older job of making pages eligible for rich results in Google Search. Treat it as housekeeping, not a lever for AI visibility.

Does the text need to be in the HTML?#

Google says it can process content in JavaScript as long as it isn't blocked, though JavaScript-heavy sites are generally more complex to optimize. OpenAI, Anthropic and Perplexity don't say in the pages we read whether their crawlers run JavaScript. We therefore can't tell you what they do, and we won't guess.

The safe habit costs little: make sure your headings, answers, prices and contact details are in the HTML the server sends, not added after load by a script. To check, open the page source (not the inspector) and search for a sentence from the page. If it isn't there, a crawler that doesn't run scripts won't see it either. Our technical SEO checklist has a step for checking what Google sees after JavaScript runs. Slow pages also lose visitors whatever else you do, which our Core Web Vitals guide covers.

Do you need an llms.txt file?#

For Google, no. Its AI guide says you don't need new machine readable files or AI text files to appear in Google Search, and that Google Search itself doesn't use them. We found no statement in OpenAI's, Anthropic's or Perplexity's crawler documentation that they read llms.txt, so we can't say any of them does. If you publish one anyway, it costs little, but it should not take time from the page itself. (Our own site publishes an llms.txt as an extra. Nothing we have read says it has any effect on rankings or answers.)

How can you limit what AI features show from your pages?#

Google's robots meta tag documentation describes four controls that apply to its search results, AI Overviews and AI Mode included:

• nosnippet stops a text or video snippet being shown for the page, and Google says it also stops the content being used as direct input for AI Overviews and AI Mode.
• data-nosnippet, on a span, div or section, keeps that part of the page out of snippets.
• max-snippet caps the snippet length, and Google says it limits how much can be used as direct input for those features, unless you have granted permission another way.
• noindex keeps the page out of search results altogether.

Each has a cost: a page with nosnippet or noindex gives up the visibility you were trying to earn. Use them for pages with content you don't want excerpted, not as a default.

How do you measure AI visibility?#

Start with Google, because it gives you numbers. Its AI features page says that sites appearing in AI features are included in your overall search traffic in Search Console, in the Performance report under the Web search type. Its AI guide also points to a separate generative AI performance report, which Search Console's help describes as showing organic impressions from Google Search generative AI features over time, the pages with the most and fewest impressions, and where they come from, such as device or country. It shows data once the site has enough impressions.

For the other products, we found no documentation of a report you can pull yourself. Look at the referral sources in your analytics for visits from the AI products' own sites, and treat the numbers as a rough count, since no company we read documents how its visits show up in analytics.

If Bing matters to your customers, look at IndexNow, which lets you tell participating search engines when a URL is added, updated or deleted. Its documentation says a 200 response only means the engine received the URL, not that it will index it, and it doesn't mention Google. We send it after deploys that add or change pages, and treat it as a courtesy notice, not a guarantee. We did not cite Microsoft's own guidance on Copilot here, because we could not read its page in full.

What do we do on ndcompany.in?#

Our robots.txt allows the answer-engine crawlers by name and blocks CCBot and Bytespider. It also carries a Content-Signal line: search=yes, ai-input=yes, ai-train=no. contentsignals.org shows how the line is written in robots.txt, with yes or no for each of three signals. Cloudflare, which introduced them, defines them in its announcement: search is building a search index and providing results (not AI summaries), ai-input is feeding content into AI models, such as for generative AI search answers, and ai-train is training or fine-tuning models.

Read that line as a statement of preference. Cloudflare says the signals are not technical countermeasures against scraping, and that some companies may ignore them. We keep it because it states in a standard place what we want, and the named crawler rules above do the work for the companies that honour them.

Where should a small business start?#

1. Open yoursite.com/robots.txt and confirm it doesn't block Googlebot, OAI-SearchBot, Claude-SearchBot or PerplexityBot by accident.
2. Decide on GPTBot, ClaudeBot and Google-Extended as a business choice, knowing that for Google-Extended the answer does not change your Search rankings.
3. In Search Console, confirm your important pages are indexed.
4. View the source of your top pages and confirm the answer, price and contact details are in the HTML.
5. Rewrite one page that only repeats what others say, adding the facts only you have, with an author and a real date.
6. Check the generative AI performance report in Search Console after a few weeks.

If you'd like a second pair of eyes on any of this, our SEO services include the technical checks above and the content work that follows them.