<\/p>
Some developers have been experimenting with bot-specific Markdown delivery as a way to reduce token usage for AI crawlers.<\/p>
Google Search Advocate John Mueller pushed back on the idea of serving raw Markdown files to LLM crawlers, raising technical concerns on Reddit and calling the concept \u201ca stupid idea\u201d on Bluesky.<\/p>
A developer posted on r\/TechSEO, describing plans to use Next.js middleware to detect AI user agents such as GPTBot and ClaudeBot. When those bots hit a page, the middleware intercepts the request and serves a raw Markdown file instead of the full React\/HTML payload.<\/p>
The developer claimed early benchmarks showed a 95% reduction in token usage per page, which they argued should increase the site\u2019s ingestion capacity for retrieval-augmented generation (RAG) bots.<\/p>
Mueller responded with a series of questions.<\/p>
\u201cAre you sure they can even recognize MD on a website as anything other than a text file? Can they parse & follow the links? What will happen to your site\u2019s internal linking, header, footer, sidebar, navigation? It\u2019s one thing to give it a MD file manually, it seems very different to serve it a text file when they\u2019re looking for a HTML page.\u201d<\/p> <\/blockquote>
On Bluesky, Mueller was more direct. Responding to technical SEO consultant Jono Alderson, who argued that flattening pages into Markdown strips out meaning and structure,<\/p>
Mueller wrote:<\/p>
\u201cConverting pages to markdown is such a stupid idea. Did you know LLMs can read images? WHY NOT TURN YOUR WHOLE SITE INTO AN IMAGE?\u201d<\/p> <\/blockquote>
Alderson argued that collapsing a page into Markdown removes important context and structure, and framed Markdown-fetching as a convenience play rather than a lasting strategy.<\/p>
Other voices in the Reddit thread echoed the concerns. One commenter questioned whether the effort could limit crawling rather than enhance it. They noted that there\u2019s no evidence that LLMs are trained to favor documents that are less resource-intensive to parse.<\/p>
The original poster defended the theory, arguing LLMs are better at parsing Markdown than HTML because they\u2019re heavily trained on code repositories. That claim is untested.<\/p>
Why This Matters<\/h2>
Mueller has been consistent on this. In a previous exchange, he responded to a question from Lily Rayabout creating separate Markdown or JSON pages for LLMs. His position then was the same. He said to focus on clean HTML and structured data rather than building bot-only content copies.<\/p>
That response followed SE Ranking\u2019s analysis of 300,000 domains, which found no connection between having an llms.txt file and how often a domain gets cited in LLM answers. Additionally, Mueller has compared llms.txt to the keywords meta tag, a format major platforms haven\u2019t documented as something they use for ranking or citations.<\/p>
So far, public platform documentation hasn\u2019t shown that bot-only formats, such as Markdown versions of pages, improve ranking or citations. Mueller raised the same objections across multiple discussions, and SE Ranking\u2019s data found nothing to suggest otherwise.<\/p>
Looking Ahead<\/h2>
Until an AI platform publishes a spec requesting Markdown versions of web pages, the best practice remains as it is. Keep HTML clean, reduce unnecessary JavaScript that blocks content parsing, and use structured data where platforms have documented schemas.<\/p> <\/div>
Generative AI,News#Googles #Mueller #Calls #MarkdownForBots #Idea #Stupid #Idea #sejournal #MattGSouthern1770242933<\/p> ","protected":false},"excerpt":{"rendered":"
Some developers have been experimenting with bot-specific Markdown delivery as a way to reduce token usage for AI crawlers. Google Search Advocate John Mueller pushed back on the idea of serving raw Markdown files to LLM crawlers, raising technical concerns on Reddit and calling the concept \u201ca stupid idea\u201d on Bluesky. What\u2019s Happening A developer […]<\/p>\n","protected":false},"author":1,"featured_media":2845,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[16],"tags":[4771,179,8913,8912,90,180,80,4504],"class_list":["post-2844","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-accessibility","tag-calls","tag-googles","tag-idea","tag-markdownforbots","tag-mattgsouthern","tag-mueller","tag-sejournal","tag-stupid"],"acf":[],"_links":{"self":[{"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=\/wp\/v2\/posts\/2844","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=2844"}],"version-history":[{"count":0,"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=\/wp\/v2\/posts\/2844\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=\/wp\/v2\/media\/2845"}],"wp:attachment":[{"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=2844"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=2844"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=2844"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}