Technical Documentation Has a Second Reader: AI Agents

Key Takeaways

1. Standard browser-based analytics does not capture a significant share of machine traffic because most AI clients do not execute JavaScript.
2. Agent traffic is growing rapidly, but the number of requests alone does not indicate documentation quality.
3. llms.txt appears to have little impact on citations in AI-generated answers, but it can help runtime agents find relevant pages faster.
4. Semantic markup, ARIA, content available in raw HTML, and clear navigation remain fundamental requirements for technical documentation for AI agents.
5. Documentation effectiveness for agents should be assessed through 404 errors, irrelevant page visits, and an agent’s ability to find the correct information rather than traffic volume alone.

Technical documentation increasingly has two audiences: humans and machine clients. This raises a practical question: what should technical documentation for AI agents look like, is llms.txt necessary, should Markdown versions be served, and how much should teams invest in emerging standards today?

Public data still does not provide a universal answer to what percentage of a particular documentation site’s readers are agents. A more practical approach is to break the problem into several parts: understand how to interpret agent traffic statistics, check whether a site is accessible to machine clients, and identify practices that already have sufficient technical justification.

Yaroslav Dvorovenko, Head of Ecosystem at Whitechain, a Layer 2 network by WhiteBIT — a crypto exchange with its own technology ecosystem and infrastructure products for digital assets — shares his perspective on technical documentation, DevRel, and who actually reads it.

Why Web Analytics Misses AI Agent Traffic

Standard web analytics counts human visitors through JavaScript: a script loads on the page, runs in the browser, and sends an event. A client that does not execute scripts is not merely “undercounted” in such a dashboard — it is effectively absent.

Most machine clients do not execute JavaScript. A joint Vercel and MERJ study conducted in late 2024 analyzed more than 500 million GPTBot requests and found no instances of JavaScript execution.

Those findings are now almost two years old, so their relevance naturally comes into question. In early April 2026, Cisco DevNet engineer Oleksii Borysenko published a preprint containing HTTP fingerprints of nine coding agents and six AI assistants on a live documentation portal. The paper has not yet been peer-reviewed, but the overall picture remains similar.

Most coding agents use lightweight HTTP clients such as curl, axios, or Go net/http and do not render pages. Exceptions include Aider and OpenCode, which operate through Headless Chromium and can execute JavaScript. Browser-based agents that operate inside a full browser form another category.

As a result, crawlers and a significant share of agents cannot see content that depends on JavaScript. Instead of relying on industry averages, this can be checked directly for a specific site:

# Check whether the page text is present in the raw server response

curl -s https://your-docs.example/quickstart | grep -c "a phrase visible in the browser"

If the command returns zero even though the phrase is visible in the browser, the content is being added through JavaScript. For a significant share of machine clients, that page effectively does not exist.

How Much Documentation Traffic Comes From AI Agents?

Recent months have produced plenty of striking figures: agents already account for the majority of traffic, bots have overtaken humans, and so on. The problem is that different studies measure different things, so their numbers cannot be directly combined or compared.

These figures use different denominators, but they point in the same general direction. Combining them into a single universal percentage would be incorrect. At the same time, multiple sources suggest that machine traffic has already approached or exceeded human traffic in certain segments.

The trajectory is even more revealing. According to Mintlify, the share of agent traffic increased from 15.2% at the beginning of 2026 to 66% in July. GitBook recorded growth from less than 10% in January 2025 to approximately 41% in December, with agents accounting for the majority of “intentional reads” by May 2026.

These are two independent vendors using different methodologies but showing similarly shaped curves. Any individual metric can be distorted by measurement methodology, but matching trends across multiple sources make the overall direction considerably more convincing.

Potential conflicts of interest also matter. Mintlify and GitBook sell documentation platforms with AI features, so rising agent traffic supports their product narratives. This does not make their data incorrect, but it does mean those figures should be considered separately from independent measurements.

Cloudflare is closer to a neutral source in this context. Its estimate is more accurately described as “around 57%,” since slightly different values appear in different publications. An important qualification is that this figure refers to HTML requests rather than all internet activity. Once apps, video, and social media are included, humans still generate roughly two-thirds of overall activity.

It is equally important to define what counts as an “agent.” The term may refer to training crawlers that need large volumes of data, search bots focused on freshness, or runtime agents looking for a specific answer while performing a task.

GitBook’s breakdown adds another important detail. ChatGPT dominates agent traffic by volume at around 54%, Meta AI contributes approximately 12%, while all coding agents combined account for less than 7%.

The typical “second reader” of documentation is therefore not necessarily a coding agent. A significant share of machine clients are assistants, which makes understanding how AI agents read documentation particularly relevant to how they generate answers for users. Cursor, Claude Code, and other coding agents represent a smaller but highly targeted audience for developer documentation.

Monthly agent rankings remain volatile. Leaders can change quickly, so a single snapshot should not be treated as a permanent market distribution.

Why AI Agent Traffic Is the Wrong Metric

Even accurately measured agent traffic does not necessarily answer the most important question: how well does the documentation work for machine clients?

A more accurate agent may generate fewer requests. Every incorrect URL and every irrelevant page it reads creates another log entry. An increase in agent requests can therefore indicate not greater documentation popularity, but greater difficulty finding the required information.

Another distortion works in the opposite direction. According to the Cisco preprint, agents can compress what would be a multi-page human session into one or two requests. Traditional metrics such as session depth, time on page, and bounce rate consequently lose much of their usual meaning.

Retries can artificially inflate the numbers, while session compression can suppress them. A more practical metric is the frequency of wasted work: how many 404 errors an agent encounters and how many irrelevant pages it reads before finding the answer.

Does llms.txt Actually Work for AI Agents?

llms.txt is a file placed at the root of a website containing a structured list of pages, designed as a map for language models. Over the past two years, numerous recommendations have linked the presence of this file with visibility in AI-generated answers. Public data does not support that relationship.

In June 2026, Ahrefs analyzed 137,210 domains. Around 28% had an llms.txt file, although the authors themselves describe this as an upper bound because their sample is more technical than the average web audience. The most striking result was that 97% of valid files received no requests during May.

Among the files that did receive requests, a significant portion of the traffic came from SEO tools simply checking whether the file existed. GPTBot and Claude Code were among the notable AI clients accessing it.

In November 2025, SE Ranking analyzed 300,000 domains. Adoption was around 10%, and the study found no correlation between the presence of llms.txt and citations in AI-generated answers. Its citation prediction model even became more accurate after the llms.txt variable was removed.

The difference between 28% in the Ahrefs study and 10% in the SE Ranking study is not necessarily contradictory. Instead, it illustrates the impact of sampling: the first figure describes a more technical audience associated with a specific product, while the second covers a broader segment of the web.

None of this means llms.txt has no purpose. Its function simply appears to differ from how it is often presented in marketing materials.

For training crawlers and search bots, the file currently has limited significance. For a runtime agent that has already arrived at a specific website with a specific question, however, llms.txt can act as a map and reduce unsuccessful attempts to guess the correct URL.

Mintlify’s benchmark included 2,400 runs across 20 documentation sites using Claude Code and Codex. Without a map, agents repeatedly attempted nonexistent URLs and read irrelevant pages. Adding a reference to llms.txt reduced error frequency by almost 90% and lowered the number of requests and tokens required.

This result still requires cautious interpretation: the benchmark was conducted by a vendor using sites belonging to its own customers.

Another practical consideration is how Markdown is served. According to the same report, the share of direct requests to .md URLs increased from 25.1% in February to 54.4% in July. Meanwhile, requests using the Accept: text/markdown header fell from 25.2% to 8.3%.

This does not mean one mechanism should be disabled. Direct .md routes already represent a significant share of machine traffic, while supporting content negotiation may require only a single additional server rule.

Both mechanisms can be checked with:

curl -sI -H "Accept: text/markdown" https://your-docs.example/quickstart | grep -i "^content-type"

curl -sI https://your-docs.example/quickstart.md | head -1

In short, llms.txt has a practical use case for AI agents documentation and API references that agents visit with specific questions. At the same time, public data does not support treating it as a tool for increasing citations in AI-generated answers.

Google, llms.txt and Lighthouse Agentic Browsing

In May 2026, Google published a guide on optimization for generative search features. One of the key points in its myths section concerns machine-readable files: they are not required for appearing in Google Search because Search itself does not use them.

Around the same time, Lighthouse — the auditing tool included with Chrome — introduced an Agentic Browsing category that, among other checks, looks for llms.txt.

At first glance, this may appear contradictory within the same company. The two approaches become compatible, however, once search visibility and agent navigation are treated as separate concerns.

Both Google’s search guide and Chrome documentation explain how AI agents read documentation, including through the accessibility tree. The main disagreement concerns llms.txt. This highlights a difference in universality: the file may help a narrow class of clients, while semantic markup applies much more broadly.

The Agentic Browsing category is best considered available but not yet stable. Its checks may change without compatibility guarantees. This matters particularly for teams planning to integrate the audit into CI.

How to Test Whether AI Agents Can Read Your Documentation

Public research reveals broad trends, but the accessibility of a specific site can be tested without internal analytics. The Agentic Browsing category in Lighthouse provides reproducible output instead of a subjective assessment.

One important limitation is that Navigation mode evaluates an individual page rather than an entire site. A more representative test therefore requires several types of pages from the same domain: the homepage, a learning section, an API reference page, and a tutorial.

Differences between pages on the same site can provide more actionable information than comparisons with unrelated websites.

npm i -g lighthouse

lighthouse https://docs.whitechain.io/ \

--only-categories=agentic-browsing \

--preset=desktop \

--output=json --output-path=./agentic.json

A reproducible GitBook experiment also provides insight into how agents actually interact with documentation. The test used a fictional API that was absent from training data and gave three coding agents the same task.

Codex read the full API reference before writing code and completed all three runs without errors. Claude Code and Cursor read only the quick-start, encountered a 422 error, and then corrected their solutions based on the error message.

For documentation authors, the ranking of individual agents is not the important part. The key question is whether the entry page allows an agent to find the correct information quickly. The number of downloaded pages alone does not answer that question.

How We’re Optimizing Our Documentation for AI Agents

Semantics and ARIA are the first priority. Google’s guide and Lighthouse documentation describe the accessibility tree as one of the key channels through which agents interact with a page. Therefore, how to optimize documentation for AI agents begins with structured, semantically clear content.

There is currently no universal measurement of the exact impact of this approach on AI agents. At the same time, semantic markup and correct ARIA provide value independently of agents because they improve accessibility for screen readers and do not depend on any particular machine-readable convention.

llms.txt and direct .md routes stay. The maintenance cost of these mechanisms is low, while available benchmarks demonstrate practical benefits for coding agents.

Quick-start should lead to the full reference. Some agents may not navigate far beyond the initial page. Critical documentation sections should therefore be reachable from it through a single logical transition.

Agentic audits belong in pre-release documentation checks. Such audits provide a concrete list of problematic elements instead of assumptions about how a machine client perceives the page.

Documentation MCP server — not yet. This area is still developing, so it should not be presented as an established part of the current infrastructure.

WebMCP — postponed, with a review trigger. Investing in an experimental API for documentation before the ecosystem stabilizes introduces additional risk. The decision can be revisited once the relevant audits stabilize or when agents begin using these tools in production.

The Cost of AI Crawler Traffic

Discussions of agent traffic often assume that growth is automatically positive. Experience from infrastructure platforms shows another side of the issue.

In July 2024, the Read the Docs team reported that a single crawler downloaded 73 terabytes of archived HTML in one month. After AI crawlers were blocked, daily traffic fell from approximately 800 GB to 200 GB.

These figures no longer describe the current state of the market, but they illustrate the scale of a potential extreme case.

The level at which the problem is addressed has also changed. In July 2025, Cloudflare began blocking AI crawlers by default for new domains and launched Pay Per Crawl. In July 2026, the model evolved into Pay Per Use, linking payment not merely to crawling a page but to the use of its content in an answer.

Starting September 15, 2026, “mixed” crawlers are also expected to be blocked by default on ad-supported pages for new customers, new websites, and the free plan.

Cloudflare supports this direction partly through crawl-to-referral ratios. The exact figures vary by period, but the differences remain substantial: from roughly fourteen crawls per referral for Google to tens of thousands for some AI crawlers.

Another notable figure from 2026 is that more than half of AI crawling across Cloudflare’s network consists of repeatedly downloading pages that have not changed.

Decisions about blocking or limiting agent traffic should therefore rely on at least two internal metrics: actual traffic volume in bytes and the cost of serving it. Without them, any assessment of the economic impact remains incomplete.

What If You Don’t Optimize Documentation for AI?

The arguments against aggressive optimization in 2026 also have a reasonable evidence base. Google explicitly states that llms.txt does not affect search visibility. Multiple studies have found no relationship between the file and AI citations, while 97% of the valid llms.txt files in one study received no requests during the measured month.

Read the Docs demonstrates that agent traffic can create significant costs. WebMCP remains experimental. Cloudflare is gradually moving toward stricter crawler controls. At the same time, much of the data about agent traffic share comes from vendors with a commercial interest in the segment’s growth.

A wait-and-see approach to unstable technologies therefore has a rational basis. WebMCP does not need to be integrated before the standard stabilizes. The absence of llms.txt also creates no demonstrated risk to search visibility.

Semantics and accessibility are different. This work does not depend on the agent ecosystem and was important long before modern AI agents emerged. Even if specific machine-readable conventions change, a correctly structured page retains its value.

AI-Friendly Documentation Checklist

The history of Stack Overflow may seem like an obvious parallel, but it does not fully describe what is happening with official documentation.

The number of questions on Stack Overflow peaked at roughly 200,000 per month in 2014. By December 2025, that number had fallen below 4,000 — almost 80% lower than a year earlier.

A peer-reviewed study in Scientific Reports compared Stack Overflow with developer communities on Reddit and found a significant decline on the former without a corresponding decline on the latter. One explanation is that the social component of communities provides additional value, while pure information exchange is much easier for language models to reproduce.

Following that logic, official documentation might also be expected to lose human traffic rapidly. However, a Read the Docs co-founder wrote in early 2025 that the platform had not seen a significant decline in traffic.

One explanation is the difference in use cases. Official documentation remains a primary source for specifications, APIs, product behavior, and current technical details. In the 2025 Stack Overflow survey, technical documentation also remained the number-one learning resource for developers, cited by 68% of respondents.

Documentation has therefore not lost its reader. Instead, it has gained another type of reader — one with different format requirements, a different cost of errors, and limited visibility in standard web analytics.

Сonclusion

1. Check whether analytics captures clients without JavaScript. If measurement depends entirely on a browser script, some machine traffic will remain invisible in dashboards.
2. Run an agentic audit across several types of documentation pages. This provides a concrete list of technical issues instead of assumptions.
3. Do not evaluate machine-readable formats solely through search visibility. For agents, 404 errors, irrelevant navigation, and the number of unnecessary requests before reaching the correct answer are more informative.
4. Check the methodology and original source before using statistics. Different adoption rates or agent traffic shares may describe different samples and different types of traffic.
5. If one foundational area needs to be prioritized, it is semantic markup and correct ARIA.

The final point does not have a universally measured effect size specifically for AI agents. Its priority comes from the underlying technical mechanism and low risk: semantics simultaneously support machine readability, accessibility, and stable content structure.

Technical documentation already has a second reader, but the word “agent” covers several types of clients with different behaviors. The next stage involves clients that do not merely read documentation but take actions based on it. Preparing for that scenario starts not with experimental standards, but with documentation that machines can reliably retrieve, correctly read, and unambiguously interpret.