<\/p>
Cloudflare launched a feature that converts HTML pages to markdown when AI systems request it. Sites on its network can now serve lighter content to bots without building separate pages.<\/p>
The feature, called Markdown for Agents, works through HTTP content negotiation. An AI crawler sends a request with The launch arrives days after Google\u2019s John Mueller called the idea of serving markdown to AI bots \u201ca stupid idea\u201d and questioned whether bots can even parse markdown links properly.<\/p> Cloudflare described the feature as treating AI agents as \u201cfirst-class citizens\u201d alongside human visitors. The company used its own blog post as an example. The HTML version consumed 16,180 tokens while the markdown conversion used 3,150 tokens.<\/p> \u201cFeeding raw HTML to an AI is like paying by the word to read packaging instead of the letter inside,\u201d the company wrote.<\/p> The conversion happens at Cloudflare\u2019s edge network, not at the origin server. Websites enable it per zone through the dashboard, and it\u2019s available in beta at no additional cost for Pro, Business, and Enterprise plan customers, plus SSL for SaaS customers.<\/p> Cloudflare noted that some AI coding tools already send the Each converted response includes an Converted responses include a The Content Signals framework, which Cloudflare announced during Birthday Week 2025, lets site owners set preferences for how their content gets used. Enabling markdown conversion also applies a default usage signal, not just a format change.<\/p> Mueller was criticizing a different practice. Some site owners build separate markdown pages and serve them to AI user agents through middleware. Mueller raised concerns about cloaking and broken linking, and questioned whether bots could even parse markdown properly.<\/p> Cloudflare\u2019s feature uses a different mechanism. Instead of detecting user agents and serving alternate pages, it relies on content negotiation. The same URL serves different representations based on what the client requests in the header.<\/p> Mueller\u2019s comments addressed user-agent-based serving, not content negotiation. <\/span>In a Reddit thread about Cloudflare\u2019s feature, Mueller responded with the same position. He wrote, \u201cWhy make things even more complicated (parallel version just for bots) rather than spending a bit of time improving the site for everyone?\u201d<\/span><\/p> Google defines cloaking as showing different content to users and search engines with the intent to manipulate rankings and mislead users. The cloaking concern may apply differently here. With user-agent sniffing, the server decides what to show based on who\u2019s asking. With content negotiation, the client requests a format and the server responds. The content is the same information in a different format, not different content for different visitors.<\/p> The practical result is still similar from a crawler\u2019s perspective. Googlebot requesting standard HTML would see a full webpage. An AI agent requesting markdown would see a stripped-down text version of the same page.<\/p> Cloudflare also added content type tracking to Cloudflare Radar for AI bot traffic. The data shows the distribution of content types returned to AI agents and crawlers, broken down by MIME type.<\/p> You can filter by individual bot to see what content types specific crawlers receive. Cloudflare showed OAI-SearchBot as an example, displaying the volume of markdown responses served to OpenAI\u2019s search crawler.<\/p> The data is available through Cloudflare\u2019s public APIs and Data Explorer.<\/p> If you already run your site through Cloudflare, you can enable markdown conversion with a single toggle instead of building separate markdown pages.<\/p> Enabling Markdown for Agents also sets the Cloudflare said it plans to add custom Content-Signal policy options to Markdown for Agents in the future.<\/p> Mueller\u2019s criticism focused on separate markdown pages, not on standard content negotiation. Google hasn\u2019t addressed whether serving markdown through content negotiation falls under its cloaking guidelines.<\/p> The feature is opt-in and limited to paid Cloudflare plans. Review the Content-Signal defaults before enabling it.<\/p> <\/div> News,Web Dev SEO#Cloudflares #Markdown #Bots #sejournal #MattGSouthern1770990005<\/p> ","protected":false},"excerpt":{"rendered":" Cloudflare launched a feature that converts HTML pages to markdown when AI systems request it. Sites on its network can now serve lighter content to bots without building separate pages. The feature, called Markdown for Agents, works through HTTP content negotiation. An AI crawler sends a request with Accept: text\/markdown in the header. Cloudflare intercepts […]<\/p>\n","protected":false},"author":1,"featured_media":3441,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[16],"tags":[89,11379,8886,90,80],"class_list":["post-3440","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-accessibility","tag-bots","tag-cloudflares","tag-markdown","tag-mattgsouthern","tag-sejournal"],"acf":[],"_links":{"self":[{"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=\/wp\/v2\/posts\/3440","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=3440"}],"version-history":[{"count":0,"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=\/wp\/v2\/posts\/3440\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=\/wp\/v2\/media\/3441"}],"wp:attachment":[{"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=3440"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=3440"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/longzhuplatform.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=3440"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Accept: text\/markdown<\/code> in the header. Cloudflare intercepts it, fetches the original HTML from the origin server, converts it to markdown, and delivers the result.<\/p> What\u2019s New<\/h2>
Accept: text\/markdown<\/code> header. The company named Claude Code and OpenCode as examples.<\/p> x-markdown-tokens<\/code> header that estimates the token count of the markdown version. Developers can use this to manage context windows or plan chunking strategies.<\/p> Content-Signal Defaults<\/h2>
Content-Signal<\/code> header set to ai-train=yes, search=yes, ai-input=yes<\/code> by default, signaling the content can be used for AI training, search use, and AI input (including agentic use). Whether a given bot honors those signals depends on the bot operator. Cloudflare said the feature will offer custom Content-Signal policies in the future.<\/p> How This Differs From What Mueller Criticized<\/h2>
New Radar Tracking<\/h2>
Why This Matters<\/h2>
Content-Signal<\/code> header to ai-train=yes, search=yes, ai-input=yes<\/code> by default. Publishers who have been careful about AI access to their content should review those defaults before toggling the feature on.<\/p> Looking Ahead<\/h2>