By Elea Yuen, Founder of Visora — AI visibility practitioner based in Hong Kong. · Published
Agent Engine Optimisation (AEO) is the practice of making a website usable by autonomous AI agents — ChatGPT agent, Claude, Perplexity Comet, Amazon Buy For Me — that browse, decide, and complete tasks on a customer's behalf. SEO ranks you for humans; GEO gets you cited in AI answers; AEO lets the agent finish the job. The organising primitive is the three-bucket bot model: training crawlers (blockable without losing AI visibility), AI search indexers (blocking these removes you from AI answers), and user-triggered agents (which ignore robots.txt by design and must be managed at the CDN layer or verified via RFC 9421 Signature-Agent). The most common agent-breakers are JavaScript-only rendering, CAPTCHA walls, token bloat, and non-semantic markup. The 100-point agent-readiness checklist spans Access (25), Readability (25), Structure (20), Interfaces (15), and Commerce (15) — the last of which is what agentic commerce protocols like ChatGPT Instant Checkout (ACP), Google's UCP, and Perplexity Buy with Pro actually read.
In this guide
The three optimisation layers: SEO (rank), GEO (get cited), AEO (let agents act)
The three AI bot buckets — and why user-triggered agents ignore robots.txt
SEO ranks you for humans. GEO gets you cited in AI answers. AEO — Agent Engine Optimisation — makes your website usable by autonomous AI agents that browse, decide, and complete tasks on a customer's behalf. It is the execution layer of AI visibility.
In one line: GEO gets you recommended; AEO lets the agent complete the transaction. From 2026 on, a growing share of your "visitors" are agents acting for customers.
The Three Optimisation Layers
SEO — rank for humans
Google and Bing blue links. The customer reads results, clicks, and does everything themselves. Success = rankings and clicks.
GEO — get cited by AI answers
ChatGPT, Perplexity, Gemini, and AI Overviews name and describe you in their answers. Success = mentions, citations, share of voice.
AEO — let AI agents act
Autonomous agents (ChatGPT agent, Claude, Perplexity Comet, Amazon Buy For Me) browse your site and COMPLETE tasks: find the price, fill the form, check out. Success = completed tasks, not impressions.
Start Here: The Three Kinds of AI Bots
Every major AI vendor (OpenAI, Anthropic, Google, Perplexity, Meta, Amazon, Apple) documents the same taxonomy. The most misunderstood fact on the modern web: user-triggered agents ignore robots.txt by design.
Training crawlers
GPTBot, ClaudeBot, Google-Extended, CCBot. They collect training data. Blocking them does NOT remove you from AI answers — it only affects future model memory. A legitimate choice, not a visibility decision.
AI search indexers
OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, BingBot. These feed live AI answers. Block these and you genuinely disappear from ChatGPT search and Perplexity.
User-triggered agents
ChatGPT-User, Claude-User, Perplexity-User, AmazonBuyForMe. A human asked the AI to do something, and the agent fetches your page to do it. These ignore robots.txt by design — you manage them at the CDN/WAF layer, or verify them cryptographically (Web Bot Auth).
What Breaks Agents on Real Websites
JavaScript-only sites
Text-mode agents make one HTTP request and read the raw HTML — they never run your JS bundle. If your prices, products, or contact form only render client-side, agents see an empty page.
CAPTCHA walls
reCAPTCHA, Turnstile, hCaptcha on entry or conversion paths stall agents completely. The fix is not removing protection — it is verifying good agents cryptographically (RFC 9421 Signature-Agent) instead of challenging everything.
Token bloat
Agents read pages into a finite context window. A 60k-token page of nav menus, popups, and script noise drowns your actual content. Lean, server-rendered pages win.
Unlabelled, non-semantic markup
Agents rely on the same signals as screen readers: one h1, logical headings, labelled form fields, real button elements. Accessibility work is now agent-readiness work.
The Agent-Ready Checklist (Scored Out of 100)
Access — 25 pts
Audit robots.txt against the full AI bot taxonomy (all three buckets), keep sitemap.xml declared, no stray noindex. Decide training vs search policy separately — they are different decisions.
Readability — 25 pts
Server-render or pre-render everything an agent must read; keep JSON-LD in the initial HTML; hold pages well under a 16k-token budget.
Structure — 20 pts
Valid JSON-LD with Organization/Product/FAQPage types, canonical and OG tags, visible dates, real internal linking.
Interfaces — 15 pts
Machine surfaces: an OpenAPI spec at /.well-known/openapi.json, markdown delivery via Accept: text/markdown, RSS. llms.txt is cheap — publish it — but an Ahrefs 2026 study found 97% of llms.txt files get zero fetches a month, so treat it as a checkbox, not a strategy.
Commerce — 15 pts
Product/Offer schema with price, currency, availability and GTIN; a checkout path reachable without JavaScript; no CAPTCHA on the conversion path. This is what ChatGPT Instant Checkout (ACP), Google's UCP, and Perplexity Buy with Pro read.
Measure It with Visora
The free Agent Readiness Checker scores your site's access and readability instantly — a robots.txt matrix across ~35 documented AI bots, no-JS readability, and token budget. The full in-platform audit adds machine interfaces and commerce readiness; the Agent Task Test watches a real browser agent attempt your key flows; and the Crawler Tracker measures which bots and agents actually visit, from your server logs.
We run the same playbook on ourselves: visoraco.com's robots.txt welcomes all three buckets, every page serves markdown to agents that ask, and a public MCP server lets AI agents call Visora's tools directly.
Common Questions
How is AEO different from GEO?
GEO wins you mentions — your name inside AI answers. AEO wins you usage — agents completing tasks on your site. One is the content layer, the other the execution layer. They compound: do GEO first, then close the loop with AEO.
Should I block AI crawlers in robots.txt?
Decide per bucket. Blocking training crawlers (GPTBot, ClaudeBot, Google-Extended) is a legitimate content-policy choice. Blocking search indexers (OAI-SearchBot, PerplexityBot) genuinely removes you from AI answers. And remember: user-triggered agents ignore robots.txt entirely.
How do I know if agents are visiting my site?
GA4 can't see bots (they don't run JavaScript), so read server logs: user-triggered agent UAs (ChatGPT-User, Claude-User, Perplexity-User) are the practical demand signal, and RFC 9421 Signature-Agent headers cryptographically confirm real agents. Visora's Crawler Tracker does exactly this.
Are agents really buying things?
Already happening: ChatGPT Instant Checkout (built on OpenAI + Stripe's ACP) is live with US Etsy sellers, Perplexity Buy with Pro does one-click checkout, and Google's UCP is backed by Shopify, Walmart, Target and others. Getting your product feed and checkout path agent-ready now is the early-mover advantage.
Go Deeper
Visora for AI Agents — the MCP server, OpenAPI spec, and markdown delivery we expose to agents ourselves
Elea Yuen is the founder of Visora, a Hong Kong-based AI visibility practice. She leads the multi-engine scan research behind the State of AI Visibility report and writes the Visora Learning Center guides. More about Visora