{"id":11440,"date":"2026-07-11T21:01:05","date_gmt":"2026-07-11T13:01:05","guid":{"rendered":"http:\/\/longzhuplatform.com\/?p=11440"},"modified":"2026-07-11T21:01:05","modified_gmt":"2026-07-11T13:01:05","slug":"ai-visibility-rankings-arent-stable-new-research-shows-its-mostly-statistical-noise-via-sejournal-mattgsouthern","status":"publish","type":"post","link":"http:\/\/longzhuplatform.com\/?p=11440","title":{"rendered":"AI Visibility Rankings Aren\u2019t Stable \u2013 New Research Shows It\u2019s Mostly Statistical Noise via @sejournal, @MattGSouthern"},"content":{"rendered":"<p><\/p> <div id=\"narrow-cont\"> <p>AI visibility tracking data isn\u2019t entirely reliable. Because generative models often produce different responses, the citation shares and rankings on your dashboard are merely snapshots of a continuously changing target, not fixed facts.<\/p> <p>A difference between you and a competitor could be genuine or just fluctuation between measurements. A new IQRush paper due for release next week (we had pre-release access) provides a method to distinguish these, showing that no fixed amount of data can definitively settle the question.<\/p> <p>The paper is by Ron Sielinski, who co-founded IQRush, who sell software that measures AI visibility the way the paper argues you should. The reason it\u2019s worth your time is that a separate team published a similar repeated-measurement finding in April, so IQRush is not the only one making this case.<\/p> <p><iframe class=\"sej-iframe-auto-height\" id=\"in-content-iframe\" scrolling=\"no\" src=\"https:\/\/www.searchenginejournal.com\/wp-json\/sscats\/v2\/tk\/Middle_Post_Text\"><\/iframe><\/p> <h2>How Much These Numbers Move<\/h2> <p>Repeatedly querying SearchGPT, Gemini, or Perplexity with the same question can produce different sources each time.\u00a0They\u2019re built to add some randomness to each response, so\u00a0each citation is just one of many possible URLs it could have pulled.\u00a0A prior paper by the same author explored this variability, showing that, for example, when testing SearchGPT on running gear, Tom\u2019s Guide made up about 9.5% of citations, while Runner\u2019s World accounted for roughly 6.0%.\u00a0On the dashboard, Tom\u2019s Guide appeared more often, but the large margin of error meant the figures overlapped.\u00a0With only one sample, it wasn\u2019t accurate to say Tom\u2019s Guide outperformed Runner\u2019s World, as the 3.5-point difference was within the margin of error.\u00a0The new paper aims to prevent this mistake by addressing a simple yet often overlooked question: How much data is needed before rankings are truly meaningful?<\/p> <h2>When A Ranking Is Worth Trusting<\/h2> <p>The answer has two parts, and both need to be true for a ranking to be reliable. First, the order must stop changing.<\/p> <p>In the beginning, rankings may change frequently as new answers are added because no site has a clear edge yet.\u00a0It\u2019s only after enough answers are collected that the top sites start to stand out clearly, allowing the order to stabilize.\u00a0Also, it\u2019s important that the top sites are well apart; if they\u2019re\u00a0very close, the ranking might not be meaningful, as a tight competition doesn\u2019t really show who\u2019s truly ahead.\u00a0The paper looks at whether the difference between the top sites is bigger than the margin of error for each.\u00a0When it is, the ranking reflects a real difference. When it isn\u2019t, it\u2019s probably just statistical noise.\u00a0Both conditions need to be true at the same time, neither alone is enough.\u00a0In 30 platform-topic tests, the number of answers needed for both conditions to be met ranged from 33 to 94, counting only answers with citations.<\/p> <p>Three out of 30 didn\u2019t reach this point even after 125 questions, all on SearchGPT, where top sites were too similar to tell apart.\u00a0There is no single cutoff applicable everywhere; what works for one platform and topic may not suit another.<\/p> <h2>We\u2019ve Been Circling This<\/h2> <p>In January, I discussed SparkToro\u2019s discovery that AI tools give a different list of recommended brands more than 99% of the time you ask the same question.\u00a0That article left one question unanswered: how many times do you need to ask before the results stabilize?\u00a0This paper offers the clearest answer I\u2019ve come across.<\/p> <p>Rand Fishkin, who led that study, shares some helpful advice.\u00a0Before spending any money on tracking AI visibility, he suggests making sure your provider \u201cshows their math.\u201d The IQRush paper is a great way to do this because it provides a simple stopping rule, so you don\u2019t have to rely solely on intuition about how many runs are enough.<\/p> <p>It also fits a run of studies SEJ has covered over the past year, each reporting AI citation numbers as if they were fixed.\u00a0This one turns around, examines the measurement itself, and asks whether those numbers are stable enough to compare in the first place.<\/p> <h2>What This Changes For Your Reporting<\/h2> <p>The number on your dashboard is just a single sample.\u00a0Before trusting it, check whether your tracker performs the same check repeatedly and reports a range, or if it pulls data once and shows a clean figure.\u00a0The clean figure can actually be a warning sign, not reassurance.<\/p> <p>A gain after a content change is easy to misinterpret.\u00a0For example, a three-point increase in your SearchGPT citation share might seem like proof that your effort paid off, but such a change can fall within the natural variability of successive runs, according to the original paper\u2019s data.<\/p> <p>To claim the win, measure before and after more than once each. A single before-and-after reading cannot separate your change from ordinary noise.<\/p> <p>The platform you are measuring changes how much data you need, and not in the way you would guess. It comes down to how much independent information each answer carries, not how many citations it hands you. Gemini piles citations onto the same handful of sites within a single answer, so many of those citations tell you the same thing. SearchGPT gives fewer citations per answer but spreads them out, so each answer carries more independent information than the raw count suggests. The same number of answers on two engines does not buy the same confidence, and a budget that settles Gemini can leave you guessing on SearchGPT.<\/p> <p>Sometimes the honest answer is that you cannot say yet. Three of the 30 tests never cleanly separated their top sites within the budget. For those, the right call is to hold, not to publish a ranking the data cannot support. A tracker that can tell you \u201cnot enough data\u201d is worth more than one that prints a confident order every time you ask.<\/p> <p>The top of the ranking is the part you can most defend. With enough answers, the leaders pull away from the middle and tail, though even they are not exact. The margins of error widen fast below the front, until neighboring positions are a coin flip, and even the top 10 were not spotless, with the typical margin of error on a top-10 site running about five positions and one in five wider than 10. Trust the leaders, treat the middle and bottom as rough, and do not report exact positions past the front of the list.<\/p> <h2>What The Paper Doesn\u2019t Prove<\/h2> <p>None of this comes from a finished, peer-reviewed study. It is a preprint built on 30 platform-topic tests across three engines, using questions generated by ChatGPT rather than real user searches, over a single stretch of collection. The exact numbers will not transfer cleanly to your topics, so treat them as the shape of the problem, not a lookup table.<\/p> <p>Those counts include only answers that carried citations, which matters most on SearchGPT, because a share of its questions return no citations at all. In one topic, 125 questions produced 104 usable answers, a 17% miss, so you would need to submit more questions than those totals suggest.<\/p> <p>The check on the method is internal, too. The paper compares a ranking it calls early against that same collection\u2019s final ranking, not against any outside ground truth. That tests whether the stopping rule is consistent with itself, which is why the matching result from the unaffiliated team does real work here. The authors of that April paper, Julius Schulte, Malte Bleeker, and Philipp Kaufmann, are researchers at the University of St. Gallen. They ran a separate dataset and reached the same verdict, that a single reading is unreliable and you have to sample an engine repeatedly to trust what it tells you.<\/p> <h2>Where This Goes<\/h2> <p>The paper stops short of the thing most people will want, which is a way to know your run budget before you start collecting. Sielinski leaves that for later work and notes that the number depends on the shape of each platform\u2019s citation pattern, so a single universal budget probably is not coming.<\/p> <p>The bigger change is that AI visibility reporting is headed the way ad and analytics reporting already went, toward numbers that carry a margin of error instead of a false decimal point. That is happening while the basic plumbing is still missing, since Search Console still won\u2019t tell you which clicks came from AI. Until it does, the job falls on you to run the check more than once and report the range, not the single number your dashboard hands you.<\/p> <h2>More Resources<\/h2> <hr\/> <p><em>Featured Image: <span class=\"MuiBox-root mui-16qd35q-centeredContent-avatarContainer\"><span class=\"MuiTypography-root MuiTypography-body1 mui-1w8ttpd-contributorLabel-linkAvatarLabel\">Stokkete<\/span><\/span>\/Shuttertstock<\/em><\/p> <\/div> <p>Generative AI,SEO#Visibility #Rankings #Arent #Stable #Research #Shows #Statistical #Noise #sejournal #MattGSouthern1783774865<\/p> ","protected":false},"excerpt":{"rendered":"<p>AI visibility tracking data isn\u2019t entirely reliable. Because generative models often produce different responses, the citation shares and rankings on your dashboard are merely snapshots of a continuously changing target, not fixed facts. A difference between you and a competitor could be genuine or just fluctuation between measurements. A new IQRush paper due for release [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":11441,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[16],"tags":[4570,90,32242,1650,4716,80,4221,14342,26426,76],"class_list":["post-11440","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-accessibility","tag-arent","tag-mattgsouthern","tag-noise","tag-rankings","tag-research","tag-sejournal","tag-shows","tag-stable","tag-statistical","tag-visibility"],"acf":[],"_links":{"self":[{"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=\/wp\/v2\/posts\/11440","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=11440"}],"version-history":[{"count":0,"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=\/wp\/v2\/posts\/11440\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=\/wp\/v2\/media\/11441"}],"wp:attachment":[{"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=11440"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=11440"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=11440"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}