Google’s John Mueller responded to a post on Bluesky that lamented all the effort being wasted on AI agent accessibility instead of focusing on the more productive activity of making sites accessible for humans.
Tactical SEO
One of the consistent aspects of the history of SEO is that practitioners tend to follow tactical trends if it’s apparent that everyone else is doing it. I guess it’s human nature.
Back in the early days there was a guy who built a directory and purchased enough high PageRank links to the home page to get it to about a PageRank of 7 (on a scale of 1-10). SEOs were practically pushing each other out of the way to hand this guy their money. He sold thousands of links from his directory, and nobody thought to check if any of the links actually made a ranking difference. It was easy to check, but nobody did.
The latest tactic is LLMs.txt and creating markdown pages for AI agent consumption. Cloudflare announced in February that their infrastructure can automatically create markdown files, largely as a way for developers to save LLM token consumption. Their announcement stated that OAI-SearchBot, OpenAI’s search crawler, was consuming markdown pages.
OpenAI’s official documentation explicitly positions its OAI-SearchBot as crawling websites and does not mention or recommend markdown files.
“OAI-SearchBot is for search. OAI-SearchBot is used to surface websites in search results in ChatGPT’s search features. Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links. To help ensure your site appears in search results, we recommend allowing OAI-SearchBot in your site’s robots.txt file and allowing requests from our published IP ranges below.”
Yes, it can crawl markdown if it’s pointed to it but that’s not what it’s out there for.
Stephanie Walter posted:
“Sad truth: we are making the web accessible for AIs, not for people.
Some sites now offer a text version for LLMs, but still skip real accessibility needs like proper heading structure, landmarks that screen reader users need.”
Google’s John Mueller responded:
“A properly made website works well for AI agents … and search engines, and LLMs, and above all, for actual people.
If you’re trying to fix accessibility issues by making a separate “agent-friendly” version, you are just building technical debt. You’ll have to redo it multiple times. Just fix it.”
AI Agents Crawl HTML
It’s a relatively trivial thing to crawl HTML. Search engines have been doing it for over thirty years. Making sites accessible for everyone makes sense, especially since it’s user behavior signals that have always driven search engine rankings. From links to users searching with a brand name, Google has consistently used user signals for ranking purposes.
So, from the perspective of SEO, it makes a load of sense to make websites accessible for everyone.
Featured Image by Shutterstock/DETHAL
News,SEO#Google #Markdown #SEO #sejournal #martinibuster1783418697
