{"id":10264,"date":"2026-06-18T18:32:29","date_gmt":"2026-06-18T10:32:29","guid":{"rendered":"http:\/\/longzhuplatform.com\/?p=10264"},"modified":"2026-06-18T18:32:29","modified_gmt":"2026-06-18T10:32:29","slug":"google-exposes-the-fundamental-flaw-of-llms-txt-via-sejournal-martinibuster","status":"publish","type":"post","link":"http:\/\/longzhuplatform.com\/?p=10264","title":{"rendered":"Google Exposes The Fundamental Flaw Of LLMs.txt via @sejournal, @martinibuster"},"content":{"rendered":"<p><\/p> <div id=\"narrow-cont\"> <p>Google\u2019s John Mueller and Martin Splitt talked about LLMs.txt and markdown, with Mueller offering a surprising fact about the original purpose of LLMs.txt and also explaining why the proposed standards are have severe shortcomings.<\/p> <h2>What Discovery Is And Why It Matters<\/h2> <p>In the context of information retrieval (search), discovery is about a search engine discovering that a specific web page exists. Discovery is a part of the overall search engine architecture.<\/p> <h3>Search Engine Architecture:<\/h3> <ol> <li><strong>Discovery<\/strong><br \/>Discovering the URL (adding it to the crawl).<\/li> <li><strong>Crawling<\/strong><br \/>Downloading and parsing the content.<\/li> <li><strong>Indexing<\/strong><br \/>The process of analyzing the raw data and storing it in a structured database optimized for retrieval.<\/li> <li><strong>Ranking<\/strong><br \/>The part that everyone\u2019s interested in.<\/li> <li><strong>Serving<\/strong><br \/>This is the last step which is serving the ranked web pages in the search results.<\/li> <\/ol> <p>The above is a simplified overview of what search is and Discovery is the very first part of the process that eventually ends with ranking and serving links to websites.<\/p> <p><iframe class=\"sej-iframe-auto-height\" id=\"in-content-iframe\" scrolling=\"no\" src=\"https:\/\/www.searchenginejournal.com\/wp-json\/sscats\/v2\/tk\/Middle_Post_Text\"><\/iframe><\/p> <p>The takeaway here is that Discovery is a critical part of getting a web page queued for crawling, indexed, ranked, and eventually shown in the search results. Without Discovery a web page is invisible.<\/p> <p><strong>Now here is why this is important:<\/strong> Discovery is not a part of the proposed LLMs.txt standard. use<\/p> <h2>Original Intent Of LLMs.txt<\/h2> <p>John Mueller said that he met one of the people responsible for creating the LLMs.txt proposal and said that the creator explained that LLMs.txt was never about making a site discoverable, it was never meant to be a part of that process.<\/p> <p>This is an important point because many site owners are spending time, money, and effort generating LLMs.txt for the purpose of getting discovered and ranked in LLMs. That means that the reason people are using LLMs.txt is in conflict with the actual purpose of LLMs.txt, which has nothing to do with Discovery.<\/p> <p><em>Mueller explained:<\/em><\/p> <blockquote> <p>\u201cSo I talked with, I think, one of the people who created that proposal a while back. And the idea was really not to create something that makes it easier for search engines or LLM systems to discover all of your content, but almost more that if an LLM already knows about your site and wants to find out what else is here, then that might be an approach.<\/p> <p>And I think the aspect of using this as a way to optimize for Discovery by AI systems or Discovery by search systems, that doesn\u2019t make any sense at all.\u201d<\/p> <\/blockquote> <p>Mueller next explained that many people are using LLMs.txt in the hope of aiding the process of Discovery despite the fact that\u2019s not the purpose of LLMs.txt.<\/p> <p>He then pivoted to the fact that LLMs.txt are inherently untrustworthy because it\u2019s a site owner saying what their site\u2019s content is about, which may or may not match what\u2019s in the actual HTML.<\/p> <p><em>He continued:<\/em><\/p> <blockquote> <p>\u201cBecause it\u2019s basically you\u2019re telling these systems, like, I have the best website ever. And here are all of the pages that everyone must go to. And you must buy all of my products or whatever you put in there.<\/p> <p>So in an LLM system, it\u2026 basically, by design, can\u2019t trust what is here as a way of differentiating between different websites.\u201d<\/p> <\/blockquote> <p>Agentic Instructions<\/p> <p>Mueller then says that some of these standards proposals could be useful for helping an AI agent, which sounds like maybe he\u2019s talking about the Web Model Context Protocol (WebMCP).<\/p> <p><em>He explained:<\/em><\/p> <blockquote> <p>\u201cIf someone is already on your website, maybe some kind of automated system is helpful. Where if it goes, I want to go to Martin\u2019s Splitt and buy a photograph, then the LLM system can go to your website and can look around, like, how do you buy a photograph? Maybe he has some guidelines for me as an agent for buying photographs. That kind of makes sense.<\/p> <p>But going off and saying, I want to buy a photograph, which website has one, the system is not going to go to your website and five others and say, who has some automated information? But rather, they\u2019re trying, going to try to find the best website\u2026\u201d<\/p> <\/blockquote> <h2>LLMs.txt Is Not About Getting Discovered By AI<\/h2> <p>Mueller circled back to how people are misconstruing LLMs.txt as a way to be discovered by AI systems.<\/p> <p><em>He reasoned about this point:<\/em><\/p> <blockquote> <p>\u201cI think from that point of view, optimizing as a way of being discovered, that doesn\u2019t make sense.<\/p> <p>But what happens when an agent is on your website? I think that also just generally seems to be an open area for discussion at the moment, in that there\u2019s LLMs.txt as a proposal. There are different JSON files and well-known file types that are in discussion.<\/p> <p>There\u2019s WebMCP, which I think tries to do something similar, where they say, well, you\u2019re on this page now, but we have a programmatic interface for this, added specific URL or a specific mechanism.<\/p> <p>I think those are then almost different discussions.\u201d<\/p> <\/blockquote> <h2>Discovery And Ranking Are Still Tied To HTML<\/h2> <p>Mueller completed his thought by underlining the point that Discovery is at the HTML level.<\/p> <p><em>He explained:<\/em><\/p> <blockquote> <p>\u201cSo the generic SEO angle of how do I find a website that sells me a photograph is almost going to be completely bound to HTML pages and normal web pages.<\/p> <p>And then if a user decides to go to a specific service, then within that service, then there is a little bit more room for maybe helping an agent or an LLM system to find the right approach.<\/p> <p>But what is interesting, of course, is lots of ideas. And none of these have basically crystallized as the one thing that everyone will use. So I\u2019m sure over the next, I don\u2019t know, half year, year, or maybe longer, it\u2019s going to take a bit. And some of these agentic systems are going to kind of unify around some standard file type or mechanism or something.\u201d<\/p> <\/blockquote> <p>Mueller wasn\u2019t pushing the WebMCP standard but if AI agents become a way that users interact with websites then it\u2019s going to be something like WebMCP and not LLMs.txt that will be useful for websites, particularly for ecommerce sites.<\/p> <p>WebMCP is the naturally better fit for ecommerce because it focuses on giving AI agents actionable capabilities, like how to filter products, how to search and identify products, aids in comparing different products, and aids AI in adding a product to a shopping cart.<\/p> <p>AI agents are able to navigate using the website HTML which was designed for humans. WebMCP makes it easier for AI agents to successfully interact with the website, something that LLMs.txt does not do.<\/p> <p>While neither LLMs.txt and WebMCP help a website get discovered by AI, neither of them was created for that purpose. The Discovery part, the first stage for ranking, all happens with HTML. If that\u2019s the case, what\u2019s your next move?<\/p> <p><strong>Listen To Google\u2019s Search Off The Record Episode 111<\/strong><\/p> <p class=\"vcont\"><iframe loading=\"lazy\" title=\"Should I use markdown for my site?\" width=\"640\" height=\"360\" src=\"https:\/\/www.youtube.com\/embed\/Vkn3R6DUJ34?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen><\/iframe><\/p> <p><em>Featured Image by Shutterstock\/Master1305<\/em><\/p> <\/div> <p>News,SEO#Google #Exposes #Fundamental #Flaw #LLMs.txt #sejournal #martinibuster1781778749<\/p> ","protected":false},"excerpt":{"rendered":"<p>Google\u2019s John Mueller and Martin Splitt talked about LLMs.txt and markdown, with Mueller offering a surprising fact about the original purpose of LLMs.txt and also explaining why the proposed standards are have severe shortcomings. What Discovery Is And Why It Matters In the context of information retrieval (search), discovery is about a search engine discovering [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":10265,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[16],"tags":[2874,3286,13292,75,4318,415,80],"class_list":["post-10264","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-accessibility","tag-exposes","tag-flaw","tag-fundamental","tag-google","tag-llms-txt","tag-martinibuster","tag-sejournal"],"acf":[],"_links":{"self":[{"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=\/wp\/v2\/posts\/10264","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=10264"}],"version-history":[{"count":0,"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=\/wp\/v2\/posts\/10264\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=\/wp\/v2\/media\/10265"}],"wp:attachment":[{"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=10264"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=10264"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=10264"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}