- AI bots and scrapers could soon overtake human users on the internet
- Publishers are battling with dwindling click-through rates
- robots.txt instructions are largely being ignored
New data has claimed AI bots are rapidly taking over web traffic, with Tollbit information showing there was a new AI bot visit for every 31 human visits in the final months of 2025, up from 1:200 at the start of 2025.
At the same time, and with humans interacting more directly with AI, human visits fell by around 5% between Q3 and Q4 2025.
And it’s clear human AI usage is responsible for this shift – while training crawls fell by around 15% between Q2 and Q4 2025, RAG bots rose around 33% and AI search indexers rose around 59%.
The way we’re accessing the Internet is fundamentally changing
OpenAI leads the way in terms of scraping – its RAG bot ‘ChatGPT-User’ was around five times more active than the second-most active bot by Meta, and around 16x higher than Perplexity’s agent.
In terms of user behavior, a separate survey cited in the report found more than one-third (37%) of active AI users now start searches with artificial intelligence (like ChatGPT or Gemini) rather than traditional search.
More alarmingly, the analysis found that robots.txt, a set of instructions to tell automated bots which parts of the page they can and cannot crawl, was ignored around 30% of the time on average, and up to 42% of the time by ChatGPT-User.
With robots.txt unenforceable and totally reliant on goodwill, the instructions have been deemed effectively obsolete.
And that’s not the only bad news for publishers, who are struggling on the traffic front. Sites without direct AI licensing deals saw click-through rates (CTR) drop around 3x between Q2 and Q4 2025. Even those with AI licensing deals aren’t immune, with CTR rates also dropping.
With all of this at play, Tollbit calls for regulators to step in to establish acceptable AI patterns and protect intellectual property.
But with AI going nowhere, it’s clear that the internet is undergoing a major shift and website owners should prepare for AI bots to become their primary readers.
Follow TechRadar on Google News and add us as a preferred source to get our expert news, reviews, and opinion in your feeds. Make sure to click the Follow button!
And of course you can also follow TechRadar on TikTok for news, reviews, unboxings in video form, and get regular updates from us on WhatsApp too.
https://cdn.mos.cms.futurecdn.net/sFJB6vZMjM2y7nZNoDeghS-970-80.jpg
Source link
