{"id":10096,"date":"2026-06-16T12:39:33","date_gmt":"2026-06-16T04:39:33","guid":{"rendered":"http:\/\/longzhuplatform.com\/?p=10096"},"modified":"2026-06-16T12:39:33","modified_gmt":"2026-06-16T04:39:33","slug":"97-of-llms-txt-files-got-no-requests-ahrefs-data-shows-via-sejournal-mattgsouthern","status":"publish","type":"post","link":"http:\/\/longzhuplatform.com\/?p=10096","title":{"rendered":"97% Of LLMS.txt Files Got No Requests, Ahrefs Data Shows via @sejournal, @MattGSouthern"},"content":{"rendered":"<p><\/p> <div id=\"narrow-cont\"> <p>Ahrefs analyzed logs from 137,000 domains and found 97% of llms.txt files got zero requests. No bots, no humans.<\/p> <p>The analysis used Ahrefs data to identify user agents fetching files. Around 28% of 137,000 domains publish an llms.txt file, but since Ahrefs\u2019 customers are more technical, the actual adoption on the broader web is likely lower.<\/p> <p>Of roughly 38,000 domains with valid files, only about 1,100 received any traffic.<\/p> <p>Of files with requests, 96% came from bots, mostly non-AI. AI retrieval bots linked to ChatGPT and Perplexity made up 1%.<\/p> <h2>Who Fetches llms.txt Files<\/h2> <p>SEO audit tools had 21% requests, then unidentified bots (14%), web crawlers like Googlebot (13%), and tech profiling tools like BuiltWith (11%).<\/p> <p>AI bots, across four categories, made up 19% of requests. AI is the largest segment, but the breakdown differs from most llms.txt advocates\u2019 expectations.<\/p> <p><iframe class=\"sej-iframe-auto-height\" id=\"in-content-iframe\" scrolling=\"no\" src=\"https:\/\/www.searchenginejournal.com\/wp-json\/sscats\/v2\/tk\/Middle_Post_Text\"><\/iframe><\/p> <p>Coding agents sent 10% of requests, training crawlers 5%, assistants 2%. Claude-Code and GPTBot were the top individual bots.<\/p> <p>Slackbot alone fetched llms.txt files more often than PerplexityBot did.<\/p> <h2>The Industry Studying Itself<\/h2> <p>The report found 12% of requests from tools that audit, scan, or study llms.txt files rather than use them.<\/p> <p>GEO and AEO readiness tools sent 5% of requests; dedicated scanners and validators sent 3%, more than AI retrieval bots and assistants combined. Research bots sent 2%, with the largest research crawler identifying as a prompt injection survey.<\/p> <p>An ecosystem has developed around scoring and cataloging a file format before a significant audience appears.<\/p> <h2>No AI Bot Looks For Files That Don\u2019t Exist<\/h2> <p>Requests for \/llms.txt paths with 404 errors drew no AI traffic. Humans hitting those 404s seem to be people typing the URL into browsers, likely checking competitors.<\/p> <p>The Chrome Lighthouse llms.txt audit, which reignited the llms.txt debate in May, generated about 22 requests across the dataset, roughly 1 in 1,000.<\/p> <h2>Why This Matters<\/h2> <p>The data lines up with what Google\u2019s John Mueller has said about llms.txt for over a year. Lily Ray pressed Mueller on the gap between Google Search\u2019s dismissal and Chrome\u2019s Lighthouse audit. He said llms.txt is \u201cnot done for search\u201d and called it a \u201ctemporary crutch, perhaps to save some tokens\u201d for AI coding tools.<\/p> <p>The data shows the file\u2019s audience is coding agents and training crawlers, not AI search and retrieval bots that would generate citations.<\/p> <p>We reported on the split between Google Search and Lighthouse documentation in May. SE Ranking\u2019s earlier analysis of 300,000 domains showed no connection between having llms.txt and AI citation frequency. Ahrefs\u2019 data points to one possible reason: the bots most directly tied to live AI retrieval barely requested these files in May.<\/p> <h2>Looking Ahead<\/h2> <p>The prompt injection finding is worth watching. Ahrefs found a crawler studying llms.txt as a prompt injection risk, since agents trust ingested content. Sites auto-generating these files via CMS should review their content.<\/p> <p>Every figure in this report is a ceiling. Ahrefs measured requests, not whether bots acted on what they fetched.<\/p> <hr\/> <p><em>Featured Image: <span class=\"MuiBox-root mui-16qd35q-centeredContent-avatarContainer\"><span class=\"MuiTypography-root MuiTypography-body1 mui-1w8ttpd-contributorLabel-linkAvatarLabel\">sdecoret<\/span><\/span>\/Shutterstock<\/em><\/p> <\/div> <p>Generative AI,News#LLMS.txt #Files #Requests #Ahrefs #Data #Shows #sejournal #MattGSouthern1781584773<\/p> ","protected":false},"excerpt":{"rendered":"<p>Ahrefs analyzed logs from 137,000 domains and found 97% of llms.txt files got zero requests. No bots, no humans. The analysis used Ahrefs data to identify user agents fetching files. Around 28% of 137,000 domains publish an llms.txt file, but since Ahrefs\u2019 customers are more technical, the actual adoption on the broader web is likely [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":10097,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[16],"tags":[8723,450,512,4318,90,22496,80,4221],"class_list":["post-10096","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-accessibility","tag-ahrefs","tag-data","tag-files","tag-llms-txt","tag-mattgsouthern","tag-requests","tag-sejournal","tag-shows"],"acf":[],"_links":{"self":[{"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=\/wp\/v2\/posts\/10096","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=10096"}],"version-history":[{"count":0,"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=\/wp\/v2\/posts\/10096\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=\/wp\/v2\/media\/10097"}],"wp:attachment":[{"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=10096"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=10096"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=10096"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}