Website optimization for AI is the technical work that makes a site readable and usable by artificial intelligence crawlers and agents: the right robots.txt for each crawler, text delivered in the HTML, structured data, llms.txt and WebMCP readiness. At Busca Orgânica, we carry out this layer with a page-by-page report.
Website optimization for AI at Busca Orgânica starts with a technical question: what does an AI crawler receive when it requests a page from your site? In many assessments, the answer is less than what the visitor sees, because robots.txt blocks the crawler, the text only appears after a script runs or the structured data arrives with errors. This work fixes that layer and also handles the next one: a site an AI agent can use, not just read.
What is website optimization for AI beyond technical SEO?
Website optimization for AI is the part of the technical work that exists because of AI crawlers and agents. Technical SEO, explained in the post technical SEO, makes sure Google crawls and indexes the site. This work checks what each AI crawler receives and what robots.txt allows each one to do. It also checks whether the text arrives without depending on JavaScript and whether the site offers actions an agent can perform.
The line between this service and its siblings is the question each one answers: here, whether the machine receives the page; in SEO for AI, whether the text deserves to be clipped; in GEO, whether the brand appears in answers, with citation tracking. The difference from traditional technical SEO comes down to three points. robots.txt now has rules per AI crawler, not just for Googlebot. The text needs to be in the HTML sent by the server, because Google runs the page's scripts and not every AI crawler does the same. And the site starts being prepared for agents that act on it, such as requesting a quote.
Which AI crawlers visit your site?
Each AI company has crawlers with different functions, and robots.txt addresses each one by name. Allowing or blocking is the client's decision; our job is to explain the effect of each rule and apply it without errors. These are the main ones from OpenAI and Google:
| Crawler | Company | What it does | Effect of blocking |
|---|---|---|---|
| GPTBot | OpenAI | Collects content that may be used to train models | The content is excluded from training; ChatGPT search depends on another crawler. |
| OAI-SearchBot | OpenAI | Shows and links sites in ChatGPT search answers | The site stops appearing as a ChatGPT search result. |
| ChatGPT-User | OpenAI | Opens pages when a user asks, during a conversation | Follows the rule OpenAI publishes for user requests. |
| Googlebot | Crawls for Search, including AI Overviews and AI Mode | The site disappears from Google Search entirely. | |
| Google-Extended | robots.txt token that controls use of content for Gemini training and grounding | Does not change presence or position in Google Search. |
The functions follow the OpenAI crawlers documentation and the Google common crawlers page. One detail confuses many people: Google-Extended is not a crawler that visits the site, just a name used in robots.txt, and the collection is still done by Google's crawlers. Other names often appear in access logs, such as ClaudeBot, from Anthropic, and CCBot, from Common Crawl.
Example: the robots.txt on our own site
On 10/01/2026, we decided to allow AI training crawlers on the Busca Orgânica site: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider and meta-externalagent. For a company that wants to be cited when the topic is SEO and GEO, allowing them makes sense; a company with paid content or sensitive data may decide the opposite. In the file, each crawler's rule takes two lines, such as User-agent: GPTBot followed by Allow: /.
The second example is a problem we often find in assessments: a site built by script. The table shows what each version delivers to a crawler that reads only the HTML sent by the server.
| What we check | Script-built site | Site with text in the initial HTML |
|---|---|---|
| Page text | An empty container and the script URL | The full text, already in the delivered code. |
| H1 and H2 headings | Missing until the script runs | Present in the code. |
| Internal links | Created later, by click or script | Regular links the crawler follows. |
| Structured data | Sometimes injected only by script | In the page code. |
| What AI receives | An almost empty page | The whole page. |
Why rendering blocks AI from reading
Rendering is one of the problems we find most often in assessments, and it almost never shows up unless you look for it. When the site builds the content in the browser, after a script runs, the visitor sees the full page, and a crawler that reads only the HTML sent by the server receives an empty container. The trap is that Search Console's URL Inspection may show the whole page, because Google runs the script before reading; the source code, which is what many AI crawlers receive, still has no text. That is why we always compare both versions, and website optimization for AI resolves the difference by delivering the essential content in the initial HTML.
The approach depends on the site's technology: server-side rendering, page pre-rendering or static generation. When the site is built through our website design service, the text already ships in the initial HTML. For sites built by other vendors, we deliver the specification to the client's developer and check the result.
llms.txt: what it is and when it is worth it
llms.txt is a Markdown file, published at the site root, that lists the main pages with a short description of each, so language models can find the essentials with less effort. It is a proposal published at llmstxt.org, not a standard adopted by search engines; the file also does not replace page content. A typical llms.txt starts with the site name as the title, a summary line and lists of links to services, service area pages and posts, each link followed by a sentence.
We create the file when the site has many pages and a structure worth summarizing, and we keep the list updated as new pages go live. What is known about who reads the file today is in the post llms.txt: what it is and when it makes sense.
WebMCP: a site AI can use
WebMCP is a proposal under discussion in the W3C Web Machine Learning community group. It lets a site declare actions that an AI agent in the browser can perform, such as searching for a product or requesting a quote, without guessing where to click. The idea is related to the Model Context Protocol (MCP), used to connect assistants to tools. The text of the proposal is at WebMCP proposal (W3C), and the concept in detail is in the post what is WebMCP.
Because it is a proposal, it is not yet something to deploy on every site. Today, laying the groundwork means having forms with well-labeled fields, actions with their own URL, such as the quote page, and structured data that describes services and products. That is what we check in the checklist below.
The technical checklist we apply
Every page template on the site goes through the same routine. The table shows the item, what we check and the tool used.
| Item | What we check | Tool |
|---|---|---|
| Crawler version | Whether the delivered HTML includes the text, headings and links the visitor sees | Page source code and Search Console URL Inspection. |
| robots.txt | Whether each AI crawler is allowed or blocked according to the client's decision | Reading the file and server access logs. |
| Rendering | Whether the content depends on a script to appear | Comparing the delivered HTML with the rendered page. |
| Structured data | Whether the schema.org types are correct and error-free | Google Rich Results Test. |
| Performance | How pages perform on Core Web Vitals, the experience metrics Google uses | PageSpeed Insights. |
| llms.txt and actions | Whether the file exists and is up to date and whether forms have labeled fields | Reading the file and the form. |
| Documentation | Whether each adjustment was recorded, with date and page | Report delivered to the client. |
How does the optimization work in practice?
Busca Orgânica is a technology company with a proprietary methodology, headquartered in Guarulhos (SP), with more than 700 clients served since 2021 (Grupo Orion corporate materials, Sep/2026). On the AI side, every project starts with a technical survey, in this sequence:
- Survey: we access each page template as a crawler and compare it with the browser version.
- Report: we list, page by page, what is missing from the crawler version, the state of robots.txt, schema errors and performance, in order of business value.
- Crawler decision: we present the effect of allowing or blocking each AI crawler, and the client decides.
- Fix: our team makes the fixes when we built the site; when we did not, the client's developer receives the specification.
- Validation: we access the site again as a crawler and confirm that the text, links and data come through.
- Record: we document every adjustment, so the work can be audited at any time.
Where does technical optimization make the most difference?
It makes the most difference on sites built with frameworks that generate the page in the browser, on catalogs with many product pages and on sites that recently switched platforms. It also matters for companies that blocked AI crawlers without deciding to, through default hosting or CDN settings (content delivery network, the service that delivers the site from servers close to the visitor), and for those who want AI agents to be able to request quotes or look up products. For a small corporate site built in plain HTML, the gain is smaller: often it is enough to check robots.txt and structured data.
The honest limit of technical optimization
Making the site readable is a necessary condition for being cited, but not a sufficient one. A technically perfect site with weak content or little authority still will not be chosen; content is handled in SEO for AI and authority in GEO. That is why the technical work is measured by what can be verified: crawlers with a decided access policy, text in the HTML, error-free schema and performance measured on Core Web Vitals. In organic search, Busca Orgânica works with a performance guarantee written into the contract, with goals agreed at the start of the project. In this service, the checklist items are the deliverables that count, and AI citations fall outside what the contract promises; the full rule is in SEO with a guarantee.
How much does website optimization for AI cost?
The cost of website optimization for AI is calculated per page template, not per page: a site with hundreds of URLs may have only five templates, such as home, service, city, post and contact, and pages sharing a template are fixed together. On top of that number, the following factors weigh in:
- The site's technology and whether the text depends on a script to appear, which may call for server-side rendering or pre-rendering.
- How many crawlers are part of the robots.txt decision and whether the hosting or CDN blocks any of them by default.
- Whether llms.txt and agent readiness are included, beyond the basics of crawlers, rendering and schema.
On a new site built by Busca Orgânica, much of this layer comes ready from day one, and the work becomes maintenance instead of renovation.
Want to see what an AI crawler receives when it visits your site, page by page? Request a technical analysis from Busca Orgânica and get a report showing what to fix first. Talk to Busca Orgânica.
