What The Latest Web Almanac Report Reveals About Bots, CMS Influence & llms.txt via @sejournal, @theshelleywalsh

The Web Almanac is an annual report that translates the HTTP Archive dataset into practical insight, combining large-scale measurement with expert interpretation from industry experts.

To get insights into what the 2025 report can tell us about what is actually happening in SEO, I spoke with one of the authors of the SEO chapter update, Chris Green, a well-known industry expert with over 15 years of experience.

Chris shared with me some surprises about the adoption of llms.txt files and how CMS systems are shaping SEO far more than we realize. Little-known facts that the data surfaced in the research, and surprising insights that usually would go unnoticed.

You can watch the full interview with Chris on the IMHO recording at the end, or continue reading the article summary.

“I think the data [in the Web Almanac] helped to show me that there’s still a lot broken. The web is really messy. Really messy.”

Bot Management Is No Longer ‘Google, Or Not Google?’

Although bot management has been binary for some time – allow/disallow Google – it’s becoming a new challenge. Something that Eoghan Henn had picked up previously, and Chris found in his research.

We began our conversation by talking about how robots files are now being used to express intent about AI crawler access.

Chris responded to say that, firstly, there is a need to be conscious of the different crawlers, what their intention is, and fundamentally what blocking them might do, i.e., blocking some bots has bigger implications than others.

Second to that, requires the platform providers to actually listen to those rules and treat those files as appropriate. That isn’t always happening, and the ethics around robots and AI crawlers is an area that SEOs need to know about and understand more.

Chris explained that although the Almanac report showed the symptom of robots.txt usage, SEOs need to get ahead and understand how to control the bots.

“It’s not only understanding what the impact of each [bot/crawler] is, but also how to communicate that with the business. If you’ve got a team who want to cut as much bot crawling as possible because they want to save money, that might desperately impact your AI visibility.”

Equally, you might have an editorial team that doesn’t want to get all of their work scraped and regurgitated. So, we, as SEOs, need to understand that dynamic, how to control it technically, but how to put that argument forward in the business as well.” Chris explained.

As more platforms and crawlers are introduced, SEO teams will have to consider all implications, and collaborate with other teams to ensure the right balance of access is applied to the site.

Llms.txt Is Being Applied Despite No Official Platform Adoption 

The first surprising finding of the report was that adoption for the proposed llms.txt standard is around 2% of sites in the dataset.

Llms.txt has been a heated topic in the industry, with many SEOs dismissing the value of the file. Some tools, such as Yoast, have included the standard, but as yet, there has been no demonstration of actual uptake by AI providers.

Chris admitted that 2% was a higher adoption than he expected. But much of that growth appears to be driven by SEO tools that have added llms.txt as a default or optional feature.

Chris is skeptical of its long-term impact. As he explained, Google has repeatedly stated it does not plan to use llms.txt, and without clear commitment from the major AI providers, especially OpenAI, it risks remaining a niche, symbolic gesture rather than a functional standard.

That said, Chris has experienced log-file data suggesting some AI crawlers are already fetching these files, and in limited cases, they may even be referenced as sources. Green views this less as a competitive advantage and more as a potential parity mechanism, something that may help certain sites be understood, but not dramatically elevate them.

“Google has time and again said they don’t plan to use llms.txt which they reiterated in Zurich at Search Central last year. I think, fundamentally, Google doesn’t need it as they do have crawling and rendering nailed. So, I think it hinges on whether OpenAI say they will or won’t use it and I think they have other problems than trying to set up a new standard.”

Different, But Reassuringly The Same Where It Matters

I went on to ask Chris about how SEOs can balance the difference between search engine visibility and machine visibility.

He thinks there is “a significant overlap between what SEO was before we started worrying about this and where we are at the start of 2026.”

Despite this overlap, Chris was clear that if anyone thinks optimizing for search and machines is the same, then they are not aware of the two different systems, the different weightings, the fact that interpretation, retrieval, and generation are completely different.

Although there are different systems and different capabilities in play, he doesn’t think SEO has fundamentally changed. His belief is that SEO and AI optimization are “kind of the same, reassuringly the same in the places that matter, but you will need to approach it differently” because it diverges in how outputs are delivered and consumed.

Chris did say that SEOs will move more towards feeds, feed management, feed optimization.

“Google’s universal commerce protocol where you could potentially transact directly from search results or from a Gemini window obviously changes a lot. It’s just another move to push the website out of the loop. But the information, what we’re actually optimizing still needs to be optimized. It’s just in a different place.”

CMS Platforms Shape The Web More Than SEOs Realize

Perhaps the biggest surprise from Web Almanac 2025 was the scale of influence exerted by CMS platforms and tooling providers.

Chris said that he hadn’t realized just how big that impact is. “Platforms like Shopify, Wix, etc. are shaping the actual state of tech SEO probably more profoundly than I think a lot of people truly give it credit for.”

Chris went on to explain that “as well-intentioned as individual SEOs are, I think our overall impact on the web is minimal outside of CMS platforms providers. I would say if you are really determined to have an impact outside of your specific clients, you need to be nudging WordPress or Wix or Shopify or some of the big software providers within those ecosystems.”

This creates opportunity: Websites that do implement technical standards correctly could achieve significant differentiation when most sites lag behind best practices.

One of the more interesting insights from this conversation was that so much on the web is broken and how little impact we [SEOs] really have.

Chris explained that “a lot of SEOs believe that Google owes us because we maintain the internet for them. We do the dirty work, but I also don’t think we have as much impact perhaps at an industry level as maybe some like to believe. I think the data in the Web Almanac kind of helped show me that there’s still a lot broken. The web is really messy. Really messy.”

AI Agents Won’t Replace SEOs, But They Will Replace Bad Processes

Our conversation concluded with AI agents and automation. Chris started by saying, “Agents are easily misunderstood because we use the term differently.”

He emphasized that agents are not replacements for expertise, but accelerators of process. Most SEO workflows involve repetitive data gathering and pattern recognition, areas well-suited to automation. The value of human expertise lies in designing processes, applying judgment, and contextualizing outputs.

Early-stage agents could automate 60-80% of the work, similar to a highly capable intern. “It’s going to take your knowledge and your expertise to make that applicable to your given context. And I don’t just mean the context of web marketing or the context of ecommerce. I mean the context of the business that you’re specifically working for,” he said.

Chris would argue that a lot of SEOs don’t spend enough time customizing what they do to the client specifically. He thinks there’s an opportunity to build an 80% automated process and then add your real value when your human intervention optimizes the last 20% business logic.

SEOs who engage with agents, refine workflows, and evolve alongside automation are far more likely to remain indispensable than those who resist change altogether.

However, when experimenting with automation, Chris warned we should avoid automating broken processes.

“You need to understand the process that you’re trying to optimize. If the process isn’t very good, you’ve just created a machine to produce mediocrity at scale, which frankly doesn’t help anyone.”

Chris thinks that this will give SEOs an edge as AI is more widely adopted. “I suggest the people that engage with it and make those processes better and show how they can be continually evolved, they’ll be the ones that have greater longevity.”

SEOs Can Succeed By Engaging With The Complexity

The Web Almanac 2025 doesn’t suggest that SEO is being replaced, but it does show that its role is expanding in ways many teams haven’t fully adapted to yet. Core principles like crawlability and technical hygiene still matter, but they now exist within a more complex ecosystem shaped by AI crawlers, feeds, closed systems, and platform-level decisions.

Where technical standards are poorly implemented at scale, those who understand the systems that shape them can still gain a meaningful advantage.

Automation works best when it accelerates well-designed processes and fails when it simply scales inefficiency. SEOs who focus on process design, judgment, and business context will remain essential as automation becomes more common.

In an increasingly messy and machine-driven web, the SEOs who succeed will be those willing to engage with that complexity rather than ignore it.

SEO in 2026 isn’t about choosing between search and AI; it’s about understanding how multiple systems consume content and where optimization now happens.

Watch the full video interview with Chris Green here:

Thank you to Chris Green for offering his insights and being my guest on IMHO.

More Resources: 

Featured Image: Shelley Walsh/Search Engine Journal