More Than Half of Web Traffic Isn't Human. A Lot of It Could Be Your Customers.
3 min read
More than half of all web traffic is no longer human. Cloudflare, which sits in front of more than 20% of the web, says bots passed people in June, making 57.5% of webpage requests, as Fortune reported. On September 30 it added a sharper number: daily requests from AI agents on its network grew more than 1,700% in a year, a figure Forbes picked up on October 1.
For businesses, the takeaway is blunt: a growing share of the visitors reading your site are software, and many of them are researching for a buyer. If those agents can't read your site, the buyer never hears your name.
The numbers: bots passed people, and agents grow fastest
The crossover came more than a year early. Cloudflare's CEO had expected it at the end of 2027.
57.5%
of webpage requests came from bots in June
+1,700%
growth in daily AI agent requests in a year
40%
biggest drop in human visits in under a year (retail, software, IT, finance)
| What changed | Figure | Period |
|---|---|---|
| Share of webpage requests made by bots | 57.5% | June 2026 |
| Daily requests from AI agents | +1,700% | Past 12 months |
| Agents that act on the web (click links, fill forms) | +7,851% | Year over year |
| Crawler requests made to train AI | 22% → 52% | Spring 2025 to June 2026 |
| Human visits in the hardest-hit industries | Down as much as 40% | Under a year |
Source: Cloudflare. The 57.5% figure is Cloudflare data via Fortune; the 7,851% figure is HUMAN Security data via Fortune.
What stands out:
- Agents are now the fastest-growing part of bot traffic, ahead of classic crawlers.
- Agent traffic follows human habits: a weekly rhythm and a dip over the summer holidays. Usually a real person sent it.
- Agents come back every time someone asks. This traffic grows with the number of questions people ask, not with how much you publish.
- Retail, software, IT services and financial services saw human visits fall as much as 40% in under a year.
Cloudflare describes agents as "additive" for businesses that learn to serve them and "extractive" for those that don't.
Why it matters: three kinds of AI visitors
Most sites treat "AI bots" as one thing. They are three different visitors, and each one needs its own decision.
| Visitor | What it does | Examples | What we advise |
|---|---|---|---|
| Training crawler | Copies pages to train future models | GPTBot, ClaudeBot | Your call. At OpenAI, this setting is separate from search. |
| Search crawler | Indexes pages so the engine can cite them | OAI-SearchBot, PerplexityBot, Claude-SearchBot | Allow. Block it and you drop out of that engine's answers. |
| Live AI visitor | Opens a page live because a person asked | ChatGPT-User, Perplexity-User, Claude-User | Allow, and make the page easy to read. |
Source: OpenAI, Perplexity and Anthropic crawler documentation.
Three mistakes follow from treating them as one:
- Blocking everything. On Cloudflare, fewer than 1% of sites block search crawlers, while 17% block training crawlers. Most owners want AI search to show them without handing over their content for training. Blocking search crawlers takes you out of the answer.
- Trusting the old dashboard. Seer Interactive warns that agentic browsers can "inflate engagement, artificially depress bounce rates." A flat traffic line can hide a real drop in people.
- Hiding the facts. Agents read text. Pricing, specs or results inside images, PDFs or click-to-open tabs may never make it into the answer.
The AEO playbook: 5 moves to make now
- Check who's visiting. Ask your web developer which of the bots above visit your site (it shows in your server logs), or check your hosting or Cloudflare dashboard. The pages they request most are the pages AI thinks matter.
- Set each switch on purpose. Allow search crawlers and live AI visitors. Decide on training separately. Cloudflare now offers separate Search, Agent and Training controls on every plan, including Free. OpenAI says changes for its search crawler take about 24 hours to apply.
Want to stay in ChatGPT search but keep your content out of OpenAI's training? These lines in your robots.txt file do it. Send them to your web developer:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /- Lead with the answer, in plain text. Put the direct answer at the top of each key page. Use real text on the page (not text inside images), clear headings, FAQs and schema markup.
- Publish the specifics. Who you serve, what it costs, how long it takes, what results you've delivered. A Cloudflare and OpenAI pilot recently brought hundreds of millions of previously unknown pages into ChatGPT's search index. Pages with prices, timelines and results give it something to cite.
- Measure presence, not just visits. Segment chatgpt.com, perplexity.ai and gemini.google.com in analytics. Each month, check whether AI engines name you for the 10 to 20 questions your buyers ask.
One caveat: not every tracker agrees on the date. Thales puts the crossover in 2023, and no single company sees the whole web. Every dataset we've seen points the same way.
What to watch next
- Agents that pay. Cloudflare is testing Pay Per Use, which pays publishers when AI uses their content, and Monetization Gateway, which lets sites charge agents per request. Both are in beta.
- "Find me, but don't train on me." Cloudflare's Disallow AI Training setting keeps a site in search while asking crawlers not to train on it. Apple, Google and Microsoft have committed to honor it.
The bottom line
The web now has two audiences: the people who buy, and the software that researches for them. If agents can't read your site, you lose buyers and your analytics won't show it. Put your key facts in plain text and let AI search crawlers in.
See your site the way AI reads it. AEO Growth's free scan checks how ready your site is for AI engines and tells you what to fix first. Run your free AI scan