<\/p>
Reuters and Time now default to blocking AI bots, allowing only approved crawlers through allowlists, Digiday reports.<\/p>
Both publishers made the decision in May, joining People Inc. and The Atlantic, which adopted similar setups within the past year.<\/p>
Reuters says the change hasn\u2019t cost it traffic, while cutting what it spends serving bots. Executives credit the added friction with helping push AI companies toward licensing talks.<\/p>
Robots.txt works only when crawlers choose to honor it. Digiday cited a Tollbit report finding that 30% of total AI bot scrapes didn\u2019t comply with explicit robots.txt permissions.<\/p>
Blocking at other levels still has teeth, the executives say. Scrapers that route around blocks pay for workarounds, and that expense is the point.<\/p>
A blocklist catches only the bots a publisher can name. People Inc. learned that switching to an allowlist increased the number of user agents it blocked from about 2,100 to more than 30,000. Lindsay Van Kirk, svp of innovation, shared the figures at an IAB Tech Lab event in late May.<\/p>
That scale matches what robots.txt data has shown for months. A BuzzStream analysis we covered in January found 79% of top news publishers block at least one AI training bot. Anthropic\u2019s crawler documentation now warns publishers about the visibility cost of blocking its search bot. In the UK, a new conduct requirement requires Google to let websites opt out of AI search features.<\/p>
Blocking by default, a setup sometimes called default-deny, changes the decision from which bots to block to which bots to let in.<\/p>
Reuters approves a bot when it offers a \u201cfair value exchange,\u201d head of Reuters Professional Josh London told Digiday. That exchange covers four kinds of value. A bot can pay for content through licensing, send traffic back, keep the site running, or support monetization.<\/p>
The result is visible in the live Reuters robots.txt file. It lists approved crawlers from Amazon, Google, Bing\/Microsoft, Yahoo, and OpenAI, then disallows other bots from most of the site.<\/p>
Crawler access has worked the same way since robots.txt was created. Every bot gets in unless a publisher names it and blocks it.<\/p>
Now Reuters and Time are reversing that default, and the People Inc. figures show why. You can\u2019t block a bot you\u2019ve never heard of.<\/p>
Blocking has costs, though. Block a crawler, and you lose whatever it was sending back, like AI search visibility or referral traffic. That\u2019s why both publishers ask what each bot gives them before letting it in. It\u2019s a question worth asking about your own robots.txt.<\/p>
The publishers are betting there\u2019s strength in numbers. One site blocking AI bots is easy to ignore. The SPUR Coalition is building shared standards for licensing and content use. It grew to 36 organizations this month after adding 30 members. Thirty-six publishers blocking together is harder to dismiss than one.<\/p>
What\u2019s less clear is who this works for. Reuters came to the table with a newswire business and licensing deals already signed. Smaller publishers face the same choice without that leverage. They can block, but blocking costs AI visibility and doesn\u2019t guarantee anyone shows up to negotiate.<\/p>
In a deep dive I wrote a few months ago, I found that the payment pools stay small relative to traditional search revenue. If deals only come in for the biggest names, default-deny could stay a big-publisher tool.<\/p>
Featured Image: Grenar\/Shutterstock<\/em><\/p> <\/div> Generative AI,News#News #Sites #Default #Blocking #Crawlers #sejournal #MattGSouthern1781057139<\/p> ","protected":false},"excerpt":{"rendered":" Reuters and Time now default to blocking AI bots, allowing only approved crawlers through allowlists, Digiday reports. Both publishers made the decision in May, joining People Inc. and The Atlantic, which adopted similar setups within the past year. Reuters says the change hasn\u2019t cost it traffic, while cutting what it spends serving bots. Executives credit […]<\/p>\n","protected":false},"author":1,"featured_media":9696,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[16],"tags":[4674,8887,17790,90,83,80,3181],"class_list":["post-9695","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-accessibility","tag-blocking","tag-crawlers","tag-default","tag-mattgsouthern","tag-news","tag-sejournal","tag-sites"],"acf":[],"_links":{"self":[{"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=\/wp\/v2\/posts\/9695","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=9695"}],"version-history":[{"count":0,"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=\/wp\/v2\/posts\/9695\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=\/wp\/v2\/media\/9696"}],"wp:attachment":[{"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=9695"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=9695"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=9695"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}