@benword Ben Word

Serving Markdown Based on Accept Headers and User Agent Detection

Bun announced they started serving their documentation as raw Markdown to Claude Code instead of rendered HTML. The results were a 10x reduction in token usage.

It’s a great idea. I immediately went to implement it on our roots.io docs, which uses Laravel routes within our WordPress site thanks to Acorn.

I’ll show an implementation in Laravel, but the concept applies to any web framework.

Getting started

Create a Laravel app and install the CommonMark package for converting markdown to HTML:

composer require league/commonmark

Creating the files

You’ll need to create a few files to get this working:

  1. Create the controller at app/Http/Controllers/DocsController.php:
<?php

namespace App\Http\Controllers;

use Illuminate\Http\Request;
use League\CommonMark\CommonMarkConverter;

class DocsController extends Controller
{
    public function show(Request $request, $slug)
    {
        $slug = basename($slug);
        $filePath = resource_path("content/docs/{$slug}.md");

        if (!file_exists($filePath)) {
            abort(404);
        }

        $markdown = file_get_contents($filePath);

        if ($this->shouldServeMarkdown($request)) {
            return response($markdown, 200)
                ->header('Content-Type', 'text/markdown; charset=UTF-8');
        }

        // Convert to HTML for regular users
        $converter = new CommonMarkConverter();
        $html = $converter->convert($markdown);

        return view('docs', ['content' => $html]);
    }
}
  1. Create a sample Markdown file at resources/content/docs/example.md:
# Example Markdown File

Howdy!

## Markdown example

This content is written in **Markdown**.
  1. Create the view at resources/views/docs.blade.php:
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Documentation</title>
</head>
<body>
    <div class="docs-content">
        {!! $content !!}
    </div>
</body>
</html>
  1. Add the route in routes/web.php:
<?php

use Illuminate\Support\Facades\Route;
use App\Http\Controllers\DocsController;

Route::get('/', function () {
    return view('welcome');
});

Route::get('/docs/{slug}', [DocsController::class, 'show']);

Now if you visit /docs/example in your browser, you’ll see the rendered HTML. But if you make a request with an Accept header requesting Markdown or with certain user agents (like LLMs), you’ll get the raw Markdown instead.

Testing it out

To test your implementation, you can use cURL with either an Accept header or a user agent:

# Request Markdown via Accept header
curl -H "Accept: text/markdown" http://localhost:8000/docs/example

# Request Markdown via user agent detection
curl -H "User-Agent: axios/1.0" http://localhost:8000/docs/example

You should see the raw Markdown returned instead of HTML. Without these headers, you’ll get the full HTML page.

Expanding user agent detection

You can expand the user agent detection to include multiple LLMs. On the roots.io docs, I fired up Claude Code, Claude Web, and Gemini CLI to see their user agents in the access logs:

I just used axios, Claude-User, and node as the substrings to check for in the user agent header. These checks are broad and may catch non-LLM clients, but serving Markdown to them for docs should be harmless.

About content negotiation

The implementation checks for Markdown requests in two ways:

  1. Accept header: Clients that explicitly want Markdown can request it using Accept: text/markdown
  2. User agent detection: We automatically serve Markdown to detected LLMs based on their user agents

The HTTP Content-Type header for Markdown content is text/markdown. This media type is defined in RFC 7763. While browsers don’t have native support for rendering Markdown, using the proper media type ensures clients can correctly identify the content format.

Check your own access logs to discover which LLM tools your users are using.

Here’s the updated detection logic:

protected function shouldServeMarkdown(Request $request): bool
{
    // Check if client explicitly requests Markdown
    $acceptHeader = $request->header('Accept', '');
    if (str_contains($acceptHeader, 'text/markdown')) {
        return true;
    }

    // Check for LLM user agents
    $userAgent = $request->header('User-Agent', '');
    $llmAgents = ['axios', 'Claude-User', 'node'];

    foreach ($llmAgents as $agent) {
        if (str_contains($userAgent, $agent)) {
            return true;
        }
    }

    return false;
}

Why this matters

This approach provides benefits for different types of clients:

For LLMs: They’re typically charged based on token usage, and HTML is very verbose compared to Markdown. By serving Markdown directly, you can significantly reduce the number of tokens processed, leading to cost savings and improved performance.

For other clients: Tools that work better with Markdown (like documentation scrapers, content management systems, etc.) can explicitly request it via the Accept header.

Note that serving Markdown means clients miss your site’s navigation, styling context, and any interactive elements. Sometimes that context is valuable, but for many documentation use cases, the raw content is sufficient.

Other resources