Markdown for Agents: Good Tool, Bad Shortcut
rabirius design
Cloudflare recently introduced Markdown for Agents, a feature that automatically converts a web page from HTML into Markdown when an AI crawler or agent requests it. On the surface, the idea makes perfect sense. Markdown is lighter, easier to skim, and strips away the design wrappers, JavaScript clutter, and navigation noise that come with modern websites. For an AI system trying to process a page quickly, that sounds incredibly helpful.
And in many technical contexts, it is. But there is a massive difference between a useful technical feature and a best practice for AI visibility.
The goal is not to create a secondary, simplified internet that only bots can see. The goal is to make your actual website easier for search engines, AI crawlers, and assistants to understand. Those are not the same thing.
What Cloudflare’s Markdown for Agents actually does
The feature allows a crawler or AI agent to request a Markdown version of a page using an HTTP Accept header. Instead of receiving the normal HTML, the agent gets a cleaner Markdown representation generated on the fly at the edge.
For developers building agents, this is highly attractive. It reduces token usage, makes documents easier to pass into an LLM, and removes visual noise the model simply does not need. Cloudflare is not wrong that AI systems benefit from cleaner inputs—anyone who has worked with retrieval systems, RAG pipelines, or agent workflows knows raw web pages can be bloated and messy.
But the AI discovery question is entirely different. The question is not whether Markdown is useful for AI systems; the answer to that is obviously yes. The real question is whether businesses should serve alternate Markdown versions of their normal web pages to improve their AI search visibility.
For most businesses, the answer is no.
The risk: translation loss and flawed representations
When an automated tool converts your HTML to Markdown on the fly, you are trusting a generic algorithm to translate your business. Because it happens in real time, you do not have to worry about manually syncing updates between two files. But you do have to worry about translation loss.
What happens when a complex pricing table gets flattened into an unreadable text block? What happens when context, layout relationships, or crucial image alt-text do not translate cleanly? Now you have two representations of the same page—one for humans, and one for machines—and you only control how one of them looks.
Furthermore, a web page is not just words. Good HTML carries deep semantic meaning. Headings dictate hierarchy, internal links show relationships, and schema markup clarifies entities. Navigation, breadcrumbs, and semantic sections all help machines understand what a page is and how it fits into the rest of the site.
When everything is flattened into plain text, some of that critical context might survive, but much of it will not. That is the part skipped in the “Markdown is cleaner” argument. Cleaner is not always better. It may remove the very signals that search engines and AI systems already know how to use.
The better practice: make the real website machine-readable
AI systems are not helpless when they see HTML. Search engines and modern AI crawlers have been parsing it for years; the entire web is built on it. The answer is not to replace it with a parallel format, but to make the HTML better.
For most businesses, the best AI visibility work still starts with the boring fundamentals. Make the content available in the HTML without hiding it behind heavy JavaScript, image-only sections, or vague marketing language. Write direct explanations of what you do, who you serve, and what problems you solve. Add appropriate schema and use descriptive internal links that clearly explain the relationships between your services, locations, and expertise.
It is not exciting, but it works because it improves the actual website. Not a bot-only version. Not a shortcut file. The real thing.
If you want a practical checklist for that work, see What AI Agents Actually Look for on a Business Website and Your Website Was Built for Google. Was It Built for AI?.
Where Markdown and AI discovery files actually belong
This does not mean Markdown is useless. It is the perfect format for documentation, developer resources, product specs, API docs, and structured knowledge bases. It is also ideal inside AI retrieval workflows where the site owner controls the pipeline. But that is fundamentally different from saying every service page or homepage needs a Markdown twin.
There is also a critical distinction between Markdown mirrors of existing pages and AI discovery files like `/llms.txt` or `/agents.md`.
A Markdown mirror says: “Here is the same page again, but in another format.” An AI discovery file says: “Here is a structured map of who we are, what this site contains, and how to interpret our business.”
Files like `llms.txt` should not be treated as replacement pages or shadow websites. They act as an index or a brief. A good AI discovery file clarifies what the business does, what terminology the brand uses, and which URLs agents should start with. They are additive. They do not replace the website; they explain it.
On this site, we also negotiate Markdown for selected public routes when a client sends Accept: text/markdown—a controlled, allowlisted path rather than an automatic mirror of every HTML page. That is a different design choice than treating edge Markdown conversion as your primary AI visibility strategy.
When DNS-level or agent-facing discovery layers matter, they still sit on top of real endpoints and clear pages—not empty placeholders. For more on that distinction, see DNS-AID and agent readiness.
The real problem is not HTML vs. Markdown
The real problem is that most websites are inherently vague. They look polished but fail to explain the business clearly. Hero sections are full of slogans, service pages are full of filler, and case studies hide the actual outcomes.
Markdown will not fix unclear positioning. It will not fix missing schema, thin service pages, or a site where the important information is buried in PDFs and generic marketing language. If AI systems cannot understand your website, the first assumption should not be that you need Markdown. The first assumption should be that your website does not explain itself well enough.
Build your website for humans first, but ensure machines can understand it through clean HTML, structured data, and strong entity signals. Then, use AI discovery files to summarize and point to the most important parts of the site.
Cloudflare’s Markdown for Agents is a great tool for specific developer workflows and documentation. But it should not be confused with a default best practice for business discoverability. The future of AI visibility will not belong to the websites with the most alternate formats.
It will belong to the websites that are the easiest to understand.
If you want help auditing structure, crawl paths, or discovery files, get in touch or browse Projects for examples of structured, machine-readable sites.