Cloudflare yesterday announced its new Markdown for Agents feature, which serves machine-friendly versions of web content alongside traditional human-facing pages.<\/p>
- Cloudflare described the update as a response to the rise of AI crawlers and agentic browsing. <\/li>
- When a client requests
text\/markdown<\/code>, Cloudflare fetches the HTML from the origin server, converts it at the edge, and returns a Markdown version. <\/li>- The response also includes a token estimate header intended to help developers manage context windows.<\/li>
- Early reactions focused on the efficiency gains, as well as the broader implications of serving alternate representations of web content.<\/li> <\/ul>
What\u2019s happening.<\/strong> Cloudflare, which powers roughly 20% of the web, said Markdown for Agents uses standard HTTP content negotiation. If a client sends an
Accept: text\/markdown<\/code> header, Cloudflare converts the HTML response on the fly and returns Markdown. The response includesVary: accept<\/code>, so caches store separate variants.<\/p>- Cloudflare positioned the opt-in feature as part of a shift in how content is discovered and consumed, with AI crawlers and agents benefiting from structured, lower-overhead text. <\/li>
- Markdown can cut token usage by up to 80% compared to HTML, Cloudflare said.<\/li> <\/ul>
Security concern.<\/strong> SEO consultant David McSweeney said Cloudflare\u2019s Markdown for Agents feature could make AI cloaking trivial because the
Accept: text\/markdown<\/code> header is forwarded to origin servers, effectively signaling that the request is from an AI agent. <\/p>- A standard request returns normal content, while a Markdown request can trigger a different HTML response that Cloudflare then converts and delivers to the AI, McSweeney showed on LinkedIn.<\/li>
- The concern: sites could inject hidden instructions, altered product data, or other machine-only content, creating a \u201cshadow web\u201d for bots unless the header is stripped before reaching the origin.<\/li> <\/ul>
Google and Bing\u2019s markdown smackdown. <\/strong>Recent comments from Google and Microsoft representatives discourage publishers from creating separate markdown pages for large language models. Google\u2019s John Mueller said:<\/p>
- \u201cIn my POV, LLMs have trained on \u2013 read & parsed \u2013 normal web pages since the beginning, it seems a given that they have no problems dealing with HTML. Why would they want to see a page that no user sees? And, if they check for equivalence, why not use HTML?\u201d<\/li> <\/ul>
And Microsoft\u2019s Fabrice Canel\u00a0said:<\/p>
- \u201cReally want to double crawl load? We\u2019ll crawl anyway to check similarity. Non-user versions (crawlable AJAX and like) are often neglected, broken. Humans eyes help fixing people and bot-viewed content. We like Schema in pages. AI makes us great at understanding web pages. Less is more in SEO !\u201d<\/li> <\/ul>
- Cloudflare\u2019s feature doesn\u2019t create a second URL. However, it generates different representations based on request headers.<\/li> <\/ul>
The case against markdown. <\/strong>Technical SEO consultant Jono Alderson said that once a machine-specific representation exists, platforms must decide whether to trust it, verify it against the human-facing version, or ignore it:<\/p>
- \u201cWhen you flatten a page into markdown, you don\u2019t just remove clutter. You remove judgment, and you remove context.\u201d<\/li>
- \u201cThe moment you publish a\u00a0machine-only representation of a\u00a0page, you\u2019ve created a\u00a0second candidate version of reality. It doesn\u2019t matter if you promise it\u2019s generated from the same source or swear that it\u2019s \u2018the same content\u2019. From the outside, a\u00a0system now sees two representations and has to decide which one actually reflects the\u00a0page.\u201d<\/li> <\/ul>
Dig deeper. Why LLM-only pages aren\u2019t the answer to AI search<\/em><\/strong><\/p>
Why we care. <\/strong>Cloudflare\u2019s move could make AI ingestion cheaper and cleaner. But could it be considered cloaking if you\u2019re serving different content to humans and crawlers? To be continued\u2026<\/p> <\/div>
Search Engine Land is owned by Semrush. We remain committed to providing high-quality coverage of marketing topics. Unless otherwise noted, this page\u2019s content was written by either an employee or a paid contractor of Semrush Inc.<\/em> <\/p>
![\"Danny](\"https:\/\/searchengineland.com\/wp-content\/seloads\/2024\/07\/Danny-Goodwin-scaled.jpeg.webp\" "\\"Cloudflare\u2019s")
<\/div> <\/p><\/div>
- Cloudflare\u2019s feature doesn\u2019t create a second URL. However, it generates different representations based on request headers.<\/li> <\/ul>
- \u201cReally want to double crawl load? We\u2019ll crawl anyway to check similarity. Non-user versions (crawlable AJAX and like) are often neglected, broken. Humans eyes help fixing people and bot-viewed content. We like Schema in pages. AI makes us great at understanding web pages. Less is more in SEO !\u201d<\/li> <\/ul>
- \u201cIn my POV, LLMs have trained on \u2013 read & parsed \u2013 normal web pages since the beginning, it seems a given that they have no problems dealing with HTML. Why would they want to see a page that no user sees? And, if they check for equivalence, why not use HTML?\u201d<\/li> <\/ul>