@benword Ben Word

Serving Markdown to AI Agents via Accept Headers

Bun announced they started serving their documentation as raw Markdown to Claude Code instead of rendered HTML. The results were a 10× reduction in token usage.

It’s a great idea. I immediately wanted to implement it on our roots.io docs.

The catch: doing it correctly — parsing the Accept header properly, ranking by q-values, setting Vary: Accept so CDNs don’t poison caches, returning 406 Not Acceptable when appropriate — turned out to be more nuanced than a quick str_contains check. I ended up building a full reference site to cover it end-to-end.

→ acceptmarkdown.com

It’s an evergreen reference for content negotiation with Accept: text/markdown:

  • Quick start — the smallest working setup: one URL, two representations, a correct Vary header.
  • How AI-ready is your URL? — paste any URL in, get a pass/fail rubric against the four things that actually matter.
  • Guides — fundamentals, parsing q-values correctly, returning 406, caching and CDN gotchas, how to generate the Markdown from your existing content.
  • Recipes — copy-paste configs for Nginx, Caddy, Apache, Cloudflare Workers, Next.js, Astro, SvelteKit, Nuxt / Nitro, Express, WordPress, Discourse, Laravel, Rails, and Django.

Post Content to Markdown WordPress Plugin

For WordPress specifically, we shipped roots/post-content-to-markdown. It converts post content to Markdown on request, handles Accept header parsing with proper q-value ranking, sets Vary: Accept, returns 406 when the client can’t be satisfied, and advertises the Markdown sibling via Link: rel="alternate" (RFC 8288) so RFC-aware crawlers can discover it without ever sending an Accept header.

Install via Composer:

composer require roots/post-content-to-markdown

Or download the zip from the GitHub releases page and upload it via Plugins → Add New → Upload Plugin in wp-admin.

Discourse to Markdown Plugin

For Discourse forums, we shipped roots/discourse-to-markdown. It serves topics, posts, categories, tags, and /latest / /top / /hot lists as Markdown via Accept: text/markdown or a .md URL suffix, with proper q-value ranking, Vary: Accept, and Link: rel="alternate" discovery on every HTML response.

Install by adding it to your app.yml and rebuilding:

hooks:
  after_code:
    - exec:
        cd: $home/plugins
        cmd:
          - git clone https://github.com/roots/discourse-to-markdown.git

cd /var/discourse
./launcher rebuild app

Why this matters

For LLMs: they’re charged by token, and HTML is verbose compared to Markdown. Serving Markdown directly cuts the token count — often 10× on documentation pages — and improves retrieval quality because agents aren’t embedding your nav, footer, and layout chrome alongside the content.

For other clients: anything that prefers structured text (documentation scrapers, RAG pipelines, syndication tools) can request it explicitly via the Accept header.

For humans: nothing changes. Browsers send Accept: text/html like they always have, and your site renders exactly like before. Content negotiation is invisible to the people you’re already serving.

Available recipes

Copy-paste configs for common servers and frameworks:

Other resources