What it does

Every route a reader hits has a Markdown equivalent:

Two ways to ask for it:

# Accept header — for programmatic clients
curl -H "Accept: text/markdown" https://discourse.roots.io/t/welcome/5

# .md URL suffix — shareable, paste-into-chat friendly
curl https://discourse.roots.io/t/welcome/5.md

Content negotiation gotchas

Rails’ Accept handling doesn’t rank text/markdown above */*, so Accept: text/markdown, */* gets you HTML. The plugin ships its own parser that ranks by specificity, the way RFC 9110 says it should.

It also sets Vary: Accept on every response. Without it, a CDN can cache the HTML for a given URL and serve it to an agent asking for Markdown, or vice versa.

Cooked, not raw

Discourse stores two versions of every post: raw (the authoring syntax, with [quote=…] blocks and :smile: shortcodes) and cooked (the rendered HTML, with oneboxes expanded, mentions linked, and quotes attributed). The plugin converts cooked to Markdown so the output matches what readers actually see.

Discourse-specific constructs are rewritten before conversion:

• <aside class="quote"> → blockquote with > [@user](post-url): attribution
• <aside class="onebox"> → blockquote with title, URL, and excerpt
• <details><summary> → blockquote with bolded summary + body
• <a class="mention">@user</a> → @user literal
• <div class="lightbox-wrapper"> → image with the full-size URL

Install

Add the plugin to your app.yml:

hooks:
  after_code:
    - exec:
        cd: $home/plugins
        cmd:
          - git clone https://github.com/roots/discourse-to-markdown.git

Then rebuild:

cd /var/discourse
./launcher rebuild app

Flip discourse_to_markdown_enabled on in Admin → Settings → Plugins and every route above starts negotiating on Accept: text/markdown (and exposes a .md suffix as a fallback).

More context at acceptmarkdown.com, including a pass/fail readiness check for your own site. If you’re on WordPress instead of Discourse, post-content-to-markdown does the same job there.

• Backlink tools like Ahrefs, Semrush, and Moz charge hundreds a month for who-links-to-whom data that’s in the domain edge file.
• Domain authority scores are a pagerank-ish function over that edge list. CC ships harmonic centrality and pagerank in its ranks file.
• Historical web archives are a paid product, but CC has ~10 years of releases sitting in S3.
• Anchor text, nofollow, and first-seen dates live in the raw HTML, which CC publishes as WARCs.

All you need is a shell and DuckDB.

A barebones example: backlinks

Here’s a gist that pulls backlinks for any domain. Run it like this:

./backlinks.sh roots.io

And you get:

┌────────────────────┬───────────┐
│   linking_domain   │ num_hosts │
├────────────────────┼───────────┤
│ github.io          │    284009 │
│ substack.com       │    199844 │
│ cloudfront.net     │     89247 │
│ godaddy.com        │     60673 │
│ google.com         │     41269 │
│ ...                │       ... │
└────────────────────┴───────────┘

What Common Crawl publishes

Every few months, Common Crawl releases a hyperlink web graph derived from its crawl. Two files do the heavy lifting:

• Domain vertices: every domain it has seen, with a stable ID and host count.
• Domain edges: every (from_domain → to_domain) link, as ID pairs.

Domains are stored reverse-labeled (io.roots instead of roots.io) so they sort hierarchically. The script handles the flip for you.

The latest release (cc-main-2026-jan-feb-mar) is around 16 GB of gzipped edges. DuckDB scans the gzip directly, so there’s no extract step and no database to load.

How the script works

The query does three things:

1. Parse the vertex file, find the ID of the target domain.
2. Scan the edge file for rows whose to_id matches that ID.
3. Join back to vertices to get the human-readable from domains, sorted by host count.

First run downloads the two files to ~/.cache/cc-backlinks/ and then scans the edges, which takes a few minutes. Subsequent runs against a different domain reuse the cache but still scan, so the time is about the same.

More than just counts

The domain graph is one slice of what CC publishes.

• Domain ranks. Same release, separate file, with harmonic centrality and pagerank for every domain. That’s your “DR” score.
• Host-level graph. Same structure, one level finer, with subdomains kept separate.
• Historical releases. Graphs going back ~10 years, so you can chart a domain’s link growth over time.
• Raw WARCs. The underlying HTML of every crawled page. Anchor text, nofollow attributes, exact linking URLs, and first-seen dates all live here.

Caveats

Common Crawl is static HTML only. Pages that render client-side are invisible to it, so domains with JS-heavy SPAs will under-count. It’s a real gap, though SPAs should really be serving usable HTML to bots anyway.

You get counts, not enrichment. The domain graph tells you who links to you. It doesn’t tell you the anchor text, the nofollow status, the first-seen date, or the exact URL of the linking page. All of that lives in the raw WARCs. Free, but you have to parse them.

Go build something

Fair caveat: Ahrefs, Semrush, and Moz run their own crawlers. Their indexes are bigger than CC’s, fresher, and can render JS, so the numbers won’t line up. For a lot of use cases that difference doesn’t matter, and what you’re paying for is the index, the UI, and anchor-text extraction on top of data that’s mostly public. The gist is ~30 lines and is a decent starting point for working with CC data.

Common Crawl has been crawling the web and giving the results away since 2008 as a nonprofit. It’s one of the most useful open datasets on the internet and nobody had to pay for it. Huge thanks to the team for keeping it free and public.

It’s a great idea. I immediately wanted to implement it on our roots.io docs.

The catch: doing it correctly — parsing the Accept header properly, ranking by q-values, setting Vary: Accept so CDNs don’t poison caches, returning 406 Not Acceptable when appropriate — turned out to be more nuanced than a quick str_contains check. I ended up building a full reference site to cover it end-to-end.

→ acceptmarkdown.com

It’s an evergreen reference for content negotiation with Accept: text/markdown:

• Quick start — the smallest working setup: one URL, two representations, a correct Vary header.
• How AI-ready is your URL? — paste any URL in, get a pass/fail rubric against the four things that actually matter.
• Guides — fundamentals, parsing q-values correctly, returning 406, caching and CDN gotchas, how to generate the Markdown from your existing content.
• Recipes — copy-paste configs for Nginx, Caddy, Apache, Cloudflare Workers, Next.js, Astro, SvelteKit, Nuxt / Nitro, Express, WordPress, Discourse, Laravel, Rails, and Django.

Post Content to Markdown WordPress Plugin

For WordPress specifically, we shipped roots/post-content-to-markdown. It converts post content to Markdown on request, handles Accept header parsing with proper q-value ranking, sets Vary: Accept, returns 406 when the client can’t be satisfied, and advertises the Markdown sibling via Link: rel="alternate" (RFC 8288) so RFC-aware crawlers can discover it without ever sending an Accept header.

Install via Composer:

composer require roots/post-content-to-markdown

Or download the zip from the GitHub releases page and upload it via Plugins → Add New → Upload Plugin in wp-admin.

Discourse to Markdown Plugin

For Discourse forums, we shipped roots/discourse-to-markdown. It serves topics, posts, categories, tags, and /latest / /top / /hot lists as Markdown via Accept: text/markdown or a .md URL suffix, with proper q-value ranking, Vary: Accept, and Link: rel="alternate" discovery on every HTML response.

Install by adding it to your app.yml and rebuilding:

hooks:
  after_code:
    - exec:
        cd: $home/plugins
        cmd:
          - git clone https://github.com/roots/discourse-to-markdown.git

cd /var/discourse
./launcher rebuild app

Why this matters

For LLMs: they’re charged by token, and HTML is verbose compared to Markdown. Serving Markdown directly cuts the token count — often 10× on documentation pages — and improves retrieval quality because agents aren’t embedding your nav, footer, and layout chrome alongside the content.

For other clients: anything that prefers structured text (documentation scrapers, RAG pipelines, syndication tools) can request it explicitly via the Accept header.

For humans: nothing changes. Browsers send Accept: text/html like they always have, and your site renders exactly like before. Content negotiation is invisible to the people you’re already serving.

Available recipes

Copy-paste configs for common servers and frameworks:

• Nginx
• Caddy
• Apache
• Cloudflare Workers
• Next.js
• Astro
• SvelteKit
• Nuxt / Nitro
• Express
• WordPress
• Discourse
• Laravel
• Rails
• Django

Other resources

• Express.js example
• Next.js example
• Static websites and Cloudflare Workers examples

Here’s how to set it up for PHP, CSS/JavaScript/TypeScript, and Go (the primary languages I’ve been working with), based on the tools I use.

Why a shell script

You could inline the command directly in settings.json, but it turns into escaping hell fast. Hook commands are JSON strings, so any shell logic has to survive JSON escaping → shell escaping → nested quoting. The inline version of the PHP example looks like:

"command": "jq -r '.tool_input.file_path // .tool_response.filePath' | { read -r f; case \"$f\" in *.php) mago format \"$f\" >/dev/null 2>&1; if ! out=$(mago lint --minimum-fail-level=warning \"$f\" 2>&1); then printf '{\"decision\":\"block\",\"reason\":%s}' \"$(printf '%s' \"$out\" | jq -Rs .)\"; fi ;; esac; } || true"

Versus a script you can read, comment, shellcheck, test standalone with echo '{"tool_input":{"file_path":"app/Foo.php"}}' | .claude/hooks/format-lint.sh, and extend without turning the command into an illegible string. The settings.json entry stays a one-liner reference: "$CLAUDE_PROJECT_DIR"/.claude/hooks/format-lint.sh.

Setup

Every example uses the same .claude/settings.json:

{
  "hooks": {
    "PostToolUse": [
      {
        "matcher": "Write|Edit",
        "hooks": [
          {
            "type": "command",
            "command": "\"$CLAUDE_PROJECT_DIR\"/.claude/hooks/format-lint.sh"
          }
        ]
      }
    ]
  }
}

And the same script structure in .claude/hooks/format-lint.sh. The only thing that changes is the case block. Make the script executable with chmod +x .claude/hooks/format-lint.sh.

PHP — mago

Mago handles both formatting and linting for PHP.

#!/usr/bin/env bash
set -euo pipefail

f=$(jq -r '.tool_input.file_path // .tool_response.filePath // empty')
[ -n "$f" ] || exit 0

block() {
  printf '{"decision":"block","reason":%s}' "$(printf '%s' "$1" | jq -Rs .)"
  exit 0
}

case "$f" in
  *.php)
    mago format "$f" >/dev/null 2>&1 || true
    if ! out=$(mago lint --minimum-fail-level=warning "$f" 2>&1); then
      block "$out"
    fi
    ;;
esac

The rest of the script is always the same — drop a different case block inside the case "$f" in ... esac from above.

JavaScript/TypeScript/CSS — oxc

oxc provides oxlint for linting and oxfmt for formatting.

  *.js|*.jsx|*.ts|*.tsx|*.css)
    npx --no-install oxfmt "$f" >/dev/null 2>&1 || true
    if ! out=$(npx --no-install oxlint --fix --deny-warnings "$f" 2>&1); then
      block "$out"
    fi
    ;;

Go — gofmt + golangci-lint

Note: golangci-lint needs a package path, not a file path — that’s why this does dir=$(dirname "$f") and passes "$dir/..." instead of just "$f".

  *.go)
    gofmt -w "$f" 2>/dev/null || true
    dir=$(dirname "$f")
    if ! out=$(golangci-lint run --fix "$dir/..." 2>&1); then
      block "$out"
    fi
    ;;

Multiple languages

Combine the case blocks into one script:

case "$f" in
  *.php)
    mago format "$f" >/dev/null 2>&1 || true
    if ! out=$(mago lint --minimum-fail-level=warning "$f" 2>&1); then
      block "$out"
    fi
    ;;
  *.js|*.jsx|*.ts|*.tsx|*.css)
    npx --no-install oxfmt "$f" >/dev/null 2>&1 || true
    if ! out=$(npx --no-install oxlint --fix --deny-warnings "$f" 2>&1); then
      block "$out"
    fi
    ;;
esac

Quick setup

Want Claude Code to set this up for you?

You can try it without installing anything: ssh quien.sh

What it does

Run quien example.com and you get an interactive tabbed interface:

• WHOIS — registrar, dates, nameservers, contacts, with relative timestamps (“2 years ago”)
• DNS — A, AAAA, CNAME, MX, NS, TXT, PTR, SOA, and DNSSEC status
• Mail — MX records, SPF, DMARC, DKIM (probes common selectors), and BIMI with VMC chain validation
• SSL/TLS — certificate details, expiry with days remaining, SANs
• HTTP — response headers from the final destination after following redirects, security headers prioritized
• SEO — indexability checks (robots.txt, canonical, sitemap), on-page analysis (title, description, headings, images), structured data (JSON-LD, Open Graph, Twitter Cards), and performance hints (compression, caching, render-blocking resources)
• Stack — CMS detection, WordPress plugin detection, JS/CSS framework detection, and external services parsed from the HTML

It also handles IP addresses. quien 8.8.8.8 shows the network owner, CIDR range, abuse contact, reverse DNS, and ASN — with PeeringDB enrichment for peering policy, traffic profile, and IX/facility counts. A BGP fallback kicks in when RDAP doesn’t include ASN data.

SEO and Core Web Vitals

The SEO tab runs local checks out of the box. If you set a CrUX API key, it also pulls Core Web Vitals field data — real-user metrics from Chrome for LCP, INP, CLS, FCP, and TTFB. Each metric shows its p75 value with a good/needs-improvement/poor rating, plus an 8–25 week trend sparkline.

export QUIEN_CRUX_API_KEY=your-api-key

You can get a free key by enabling the Chrome UX Report API in the Google Cloud Console. Not all domains have field data — a site needs enough Chrome traffic to be included.

JSON output

For scripting, use --json for full output or run individual subcommands:

quien --json example.com
quien dns example.com
quien mail example.com
quien seo example.com

Available subcommands: whois, dns, mail, tls, http, seo, stack, all.

How it works

quien uses RDAP as its primary lookup protocol — it’s the modern replacement for WHOIS that returns structured JSON instead of freeform text. An IANA bootstrap file maps TLDs to their RDAP servers, covering ~1,200+ TLDs out of the box. For TLDs without RDAP, it falls back to traditional WHOIS with automatic server discovery via IANA referral.

The tech stack detection fetches the page HTML and inspects it for known patterns — WordPress is identified through signals like wp-includes, REST API links, and Gutenberg block markers.

Every lookup retries automatically with exponential backoff. The TUI is built with bubbletea and lipgloss. It auto-detects your terminal background for light/dark theming (override with QUIEN_THEME=light or QUIEN_THEME=dark).

Install

For Homebrew, APT, AUR, or Go install options, see the README on GitHub.

Or try it without installing: ssh quien.sh

So I built SomaFM Player, a native macOS menu bar app that gives you quick access to every SomaFM station from your status bar.

Features

• All 30+ SomaFM stations — top 10 by listener count first, then the rest alphabetically
• Live track updates — see what’s currently playing, updated every 10 seconds
• Media key support — play/pause with your keyboard media keys
• Independent volume control — adjust the app volume without changing system volume
• Clickable track titles — click to search for the current song
• Auto-play on launch — picks up where you left off
• Dark mode support — looks right on any macOS appearance
• Lightweight — lives in the menu bar, never touches the dock

Install

Homebrew:

brew tap retlehs/tap
brew install --cask somafm-player

Or download the latest release directly from GitHub Releases.

How it’s built

The app is written in Swift using native Cocoa — no Electron, no SwiftUI, just NSStatusBar, NSMenu, and AVPlayer. It has zero external dependencies.

Architecture. The app follows MVVM with protocol-based dependency injection. A SomaFMService fetches channel data from SomaFM’s JSON API, an AudioPlayer wraps AVPlayer for streaming, and a StatusBarViewModel ties everything together with Combine. All UI updates flow reactively through @Published properties.

Streaming. The app selects the highest quality playlist available from the SomaFM API and streams it through AVPlayer. It handles audio device changes gracefully — switch your headphones and playback continues.

Resilience. Network requests use exponential backoff with jitter (1s, 2s, 4s, up to 8s) and retry up to 3 times. The app differentiates between retryable errors like network failures and non-retryable ones like bad URLs, so it doesn’t waste time retrying things that will never work.

Image caching. Station artwork is cached in memory with a 100-image, 50MB limit. Images are resized to display size on download to keep memory usage low. Concurrent requests for the same image are coalesced so you don’t end up with duplicate downloads.

Persistence. A custom @UserDefault property wrapper stores volume, auto-play preference, and last played channel in UserDefaults — simple and reliable.

The entire app is about 1,800 lines of Swift. The build and release pipeline uses GitHub Actions for testing, code signing, notarization, and automatic Homebrew cask updates on new tags.

I built gh-actions, an agent skill that teaches your agent GitHub Actions best practices so you don’t have to fix the same things every time.

What it does

• Version lookup at runtime — instead of hardcoding versions that go stale, the skill tells agents to check gh api repos/{owner}/{action}/releases/latest before writing a workflow
• Security — least-privilege permissions, expression injection prevention, fork/secret safety, and SHA pinning for third-party actions (with pinact for automation)
• Caching — prefer the built-in cache input on setup actions over separate actions/cache steps
• Common patterns — concurrency groups, matrix strategies, reusable workflows, path filtering, and timeout-minutes

Why bother

LLMs are trained on a snapshot of the internet. GitHub Actions moves fast — major versions bump, best practices shift, and new features like built-in caching get added. The skill fills the gap between what the model learned and what’s current.

The version lookup approach is the key part. Instead of maintaining a static list that rots, the skill teaches the agent how to check — so it stays current without any maintenance.

Install

npx skills add retlehs/gh-actions

Works with Claude Code, Cursor, Codex, and 30+ other agents.

Also check out gh-fetch — a skill that tells agents to use the gh CLI instead of web fetching when you share GitHub URLs.

To enable it, add this to your Ghostty config:

keybind = global:cmd+grave_accent=toggle_quick_terminal
quick-terminal-size = 65%

Press cmd + ` to toggle it open and closed. The session persists between toggles.

Full docs: toggle_quick_terminal

(The ANSI art in the video is from ansimotd, a CLI that displays a random piece of ANSI art when you open a terminal. Read more about it here.)

I released Roots in 2011 as a WordPress starter theme. The starter theme eventually got renamed to Sage, and “Roots” became the name for the full ecosystem:

• 2011 — Sage (originally “Roots”), a WordPress starter theme
• 2013 — Bedrock, a WordPress boilerplate with modern dependency management
• 2014 — Trellis, server provisioning and deployment for WordPress
• 2017 — Acorn, a Laravel integration for WordPress plugins and themes

In 2013 we were still just a starter theme and didn’t have a GitHub organization, but we already had bigger plans in place. The @roots username appeared to be unused — it belonged to a guy named Rick. There was another open source project called “roots” too, a static site generator. Their team also wanted the username, and they opened an issue to coordinate getting it.

Their approach was to contact GitHub support to invoke the name squatting policy. I had actually already done this just days before them. GitHub told me the account wasn’t inactive, but they offered to pass along a message to the owner on my behalf. Here’s what I sent on June 24th, 2013:

Hey there, I run a project called Roots (github.com/retlehs/roots) and wanted to create a GitHub org to use as an umbrella for related projects. I was wondering if you’d be willing to rename your account so that we could use ‘roots’ as the organization name. If not, no worries, I understand :)

Meanwhile, the other team was pinging the account owner repeatedly and one collaborator suggested they “just harass them on twitter”. They were also shitting on our project:

And yeah, this repo is smaller in terms of stars, but the scope of the project is much larger and encompasses several repos, which is why we want a team to put them all in.

Yeah, I’ve used the roots WordPress starter theme, and I do freelance developing for WordPress themes. Thus, I know exactly what the scope of your project is.

I commented that they had no idea about the scope of our project, and that being dicks about it probably doesn’t help anything.

Rick responded:

Finally, Ben, thanks. If you guys simply asked me instead of contacting Github first, spamming me and “being dicks about it”, I would have been more than happy to help.

I’ll contact Github support and get this sorted. Good luck on the project, perhaps i’ll use it some day :).

Today the @roots organization has 40+ active public repos, 30+ packages on Packagist with over 84 million downloads, and more than 28k GitHub stars. Sage is also currently the 45th most starred PHP repository on GitHub.

Thank you again, Rick!

We’ve been building open source WordPress tools at Roots since 2011. Our roots/wordpress Composer package has nearly 20 million downloads, and a large portion of the modern WordPress Composer workflow already runs through Roots tooling. Building a Composer repository felt like a natural next step.

So we built WP Packages — an independent, open source Composer repository for WordPress plugins and themes.

Why it’s faster

WPackagist uses Composer’s older provider-includes protocol, which forces Composer to download large index files containing metadata for thousands of packages before it can resolve your dependencies.

WP Packages uses Composer v2’s metadata-url protocol, which lets Composer fetch metadata for only the packages it needs.

Cold resolve, no cache.

What else is different

• Update frequency — packages update every 5 minutes (WPackagist is ~1.5 hours)
• Package metadata — includes author, description, homepage, and support links (WPackagist has been missing these since a 2020 feature request)
• Cleaner naming — wp-plugin/* and wp-theme/* instead of wpackagist-plugin/*
• CDN-cached — metadata is served from Cloudflare R2/CDN
• Fully open source — the entire project including deployment config is on GitHub

Switching

Replace the WPackagist repository in your composer.json:

{
  "repositories": [
    {
      "type": "composer",
      "url": "https://repo.wp-packages.org"
    }
  ]
}

Then update your package names from wpackagist-plugin/* to wp-plugin/* and wpackagist-theme/* to wp-theme/*. There’s also a migration script in our GitHub repo that handles this automatically.

How it’s built

WP Packages is a single Go binary backed by SQLite. A pipeline runs every 5 minutes that discovers packages from the WordPress.org SVN repository, fetches metadata from the WordPress.org API, and builds Composer metadata files that get deployed to Cloudflare R2.

The project is community-funded through GitHub Sponsors.

Full comparison: WP Packages vs WPackagist