Your API documentation is being read by AI. The trouble is, the AI may not actually be able to read it.

The Invisible Readers

Something strange is happening. Traffic to developer docs hasn't dropped, yet time on page is cratering, scroll depth sits at zero, and tutorial completion rates have hit the floor. Stare at Google Analytics all you want—the cause won't show up.

The cause is simple: the reader has changed. It's no longer humans consuming your docs—it's AI agents. Claude Code, Cursor, and Aider pull your documentation with a single GET request and process it in 400 milliseconds. No scrolling, no clicking, no following links. Nothing registers in your funnel.

The real problem comes next. When an agent fetches your docs, a well-structured document means the task succeeds. If it isn't, the agent quietly makes things up. Code that calls your API incorrectly gets planted in developer projects around the world—and you have no way of knowing.

Google's Addy Osmani gave this problem a name: AEO, Agentic Engine Optimization. Just as we once learned SEO, it's time to learn how to optimize for AI agents.

Remember SEO?

In the mid-2000s, there was a time when people thought building a website was the whole job. "Make good content and people will find it." Not exactly wrong—but reality played out differently. If you didn't understand how search engines crawled, indexed, and ranked your site, even great content got buried on page three of the search results.

So we learned SEO. We added meta tags, built sitemaps, managed robots.txt, and optimized page load times. The substance of the content didn't change. We changed the packaging to fit how the actual consumer—the search engine—consumed it.

AEO follows exactly the same logic. Only the consumer is different. AI coding agents instead of search crawlers. Token economics instead of click patterns. llms.txt instead of sitemaps.

Agents Read Nothing Like You Think They Do

A human lands on the docs homepage, browses the menu, skims the headings, tries the code samples, follows links, and spends four to eight minutes. Every step of that journey gets captured as client-side events.

An agent is done in one or two HTTP requests. The entire documentation hierarchy a human would click through for five minutes gets pulled down in a single GET. The whole concept of a "user journey" collapses into one line in a server log. Scroll depth, dwell time, button clicks, tutorial progress—all of those metrics become meaningless.

Research data has even identified each agent's HTTP fingerprint. Claude Code identifies itself as axios/1.8.4, Cursor as got, Cline as curl/8.4.0. Check your server logs and you can see just how much AI traffic you already have. Most teams are sitting on this data—they just haven't classified it.

A 190,000-Token Document Might as Well Not Exist

The first thing an agent does after fetching a document is count tokens. Most agents effectively operate within a window of 100,000 to 200,000 tokens.

Cisco's firewall API quickstart guide runs 193,217 tokens. That one document consumes an agent's entire context window. From the agent's perspective, this document is unreadable. It will silently truncate it, skip it entirely, or wing it with whatever it half-remembers. All bad outcomes.

Token count has become a first-class metric of documentation accessibility—just as page load speed once influenced search rankings in SEO. The way slow pages slipped down the results, heavy documents get ignored by agents.

The practical guidelines look like this: quickstart pages under 15,000 tokens. API reference pages under 25,000. Concept guides under 20,000, with details split off into separate, linked pages. For a rough token estimate, divide the character count by four.

Three Files That Give Agents a Map

Just as SEO had sitemaps and robots.txt, AEO is developing its own file system for guiding agents.

robots.txt — The front door. It's the first thing an agent checks. If AI crawlers are blocked here, agents can't even reach your docs, no matter how good they are. No traffic drop, no error message—you simply become invisible. Teams really are blocking the Anthropic, OpenAI, Google, and Perplexity crawlers without meaning to. It's a ten-minute check, and skipping those ten minutes turns your entire documentation into a ghost as far as AI is concerned.

llms.txt — A sitemap for AI. A Markdown file placed at the domain root that tells agents how your documentation is structured: which pages contain what, and how many tokens each one runs. It's the signpost that lets an agent find exactly the document it needs without crawling your whole site. llms.txt itself should stay under 5,000 tokens—the signpost shouldn't weigh more than what it points to.

skill.md — A capability declaration. If llms.txt tells agents "where the docs are," skill.md declares "what this product can do." An agent can judge whether your API fits its task before reading any of the prose documentation. Filtering becomes possible before any context budget gets spent.

Making the Content Itself Agent-Friendly

Get the file system in place and you'll be discovered. But what gets discovered has to be in a form agents can digest.

Serve Markdown. Strip away the HTML and the tags, navigation, sidebars, and footers are all noise. Direct Markdown access cuts token overhead dramatically. A growing number of platforms let you append .md to a URL to reach the raw Markdown.

Put the essentials in the first 500 tokens. What this page is, what it lets you do, and what you need to get started. Agents have no patience for背景 throat-clearing—humans don't either, frankly, but agents physically weight the beginning of a document more heavily.

Use tables. Writing a parameter reference as prose burns tokens. The same information organized into a table is far more token-efficient.

Add a "Copy for AI" button. When developers use an AI assistant in their IDE, they copy-paste documentation. Copy rendered HTML and the navigation, footer, and ads come along for the ride. One button that puts clean Markdown on the clipboard transforms the quality of the context the agent receives. Anthropic and Cloudflare have already shipped this feature.

AGENTS.md Is the New README

If README.md was the entry point for human developers, AGENTS.md is becoming the entry point for AI agents. When a coding agent opens a project, it reads the AGENTS.md at the root and treats it as the guide for everything that follows.

Cisco DevNet already includes AGENTS.md by default in its open-source GitHub templates. Create a new project and it comes with an AGENTS.md pre-filled with the project structure, API documentation links, test environment, and coding conventions.

This signals a shift in development culture. The era of writing only "docs for humans" at the start of a project is ending. Writing "docs for AI" alongside them is becoming the default.

What This Means for Developers and PMs

AEO isn't just a job for technical writers.

For developers building APIs. If agents can't read your API docs, they'll recommend a competitor's API instead. Just as search ranking meant traffic in the SEO era, agent accessibility now means developer adoption.

For PMs. The definition of developer experience (DX) is expanding. It now has to cover not just human developers but AI agents too. An agent successfully digesting your docs and generating correct code is the new DX metric.

For founders. Becoming the API that AI agents recommend is the new engine of market share. Getting agents to understand your API, integrate it successfully, and come back again—that's not a marketing problem. It's a documentation-structure problem.

Actionable First Steps

This doesn't require a major infrastructure investment. The start is simple.

Open your robots.txt and check whether AI crawlers are blocked—ten minutes. Create an llms.txt and place it at your domain root—a few hours is plenty. Measure the token count of your key documentation pages and split anything over 30,000 tokens. Write a skill.md for your three core APIs. Start classifying AI agent traffic in your server logs.

As with SEO, the early movers win. And as with SEO, optimizing for agents makes your docs better for humans too. Clear structure, no filler, the essentials up front—the qualities of a good document turn out to be remarkably similar for agents and people alike.