The symptom was small. The drift was not.

It started with one modest question: can we actually publish a blog post? The machinery looked healthy from a distance — a Cloudflare Worker API, GitHub Actions for scheduling, Cloudflare Pages for the site, a changelog timeline that was supposed to narrate every merged PR across the ecosystem. But the moment I tried to push a post through, the floor gave way. What looked like a single broken button turned out to be a whole stack that had quietly drifted off its rails over the preceding months. This is the story of dragging it back.

The honest headline: nothing was catastrophically broken, and that was the problem. Catastrophes get noticed. Drift doesn’t. Tokens expire, a file format changes, a workflow starts failing at 6am every day, a repo gets mirrored over the top of another — and because no human is watching the parts that are supposed to run themselves, the rot stays invisible until you reach for the thing and it isn’t there.

Strategy 1: evidence over memory

The first real decision was philosophical: stop trusting what we remembered about the system and start proving it. Two blog repos existed — fabric-forge/blog and ry-ops/blog — with identical contents, and nobody could say which one actually deployed the live site. Memory said one thing; a stale doc said another.

So we ran an A/B test: commit a uniquely-named throwaway post to each repo, then watch which slug rendered on the live site. One showed up; one didn’t. Definitive. fabric-forge/blog is the source of truth, force-pushed down to ry-ops/blog as a mirror — which immediately explained a later mystery (writes to the mirror were being silently clobbered). A five-minute experiment settled a question that guesswork had been dodging for weeks.

The same discipline applied when I confidently asserted something about a third-party API and got corrected with a screenshot. Lesson re-learned: when you can check, check. Don’t narrate from memory and call it diagnosis.

The continuity of errors

This is the part worth dwelling on, because it’s the texture of real systems work. Almost nothing failed once. Each fix peeled back a layer and revealed the next failure underneath:

  • Blog publish stalled at “generating hero.” The pipeline ran the hero-image generation — an ~8,000-token language-model call producing a full animated SVG — inside a Cloudflare Worker. Workers have a short execution budget. The isolate was being killed mid-generation, leaving the job frozen with no error. Not a bug in the code; the wrong place to run the work.
  • Moved it to GitHub Actions → 403. The GitHub API rejected our calls because they had no User-Agent header. A one-line omission that had simply never been exercised before, because publish had never gotten far enough to make the call.
  • Fixed the header → 401. “Bad credentials.” The token had expired months earlier. The automation had been quietly unauthenticated since spring.
  • The changelog had its own stack of nested failures. Its daily job was dying at npm ci (a lockfile issue). Switching to npm install got past that — and revealed a private dependency the runner couldn’t fetch over SSH. And even if it had run, it was writing entries to the mirror repo that the force-push would erase. And the token it used was scoped to the wrong org. And — the final indignity — its repo-list parser expected flat owner/repo lines while the config file had quietly become structured YAML, so it mis-read every entry and reported “no activity.”

Six or seven distinct faults, each masking the one behind it. You don’t fix a system like this in a single stroke; you peel the onion, and you keep a steady hand while each new layer makes it briefly look worse.

Strategy 2: put the work on the right tier

The unifying technical lesson kept repeating: long, heavy work does not belong in a request-scoped Worker. The hero generation, the video render for shorts, the blog commit-and-poll — all of it wanted a runtime with no execution ceiling. The answer, every time, was GitHub Actions: the Worker validates and dispatches, and a runner does the slow, expensive part to completion. We’d already applied this pattern to scheduled dispatch and to short-form rendering; today it claimed the blog publish and the changelog too. Consistency is its own kind of reliability — when every long job lives in the same place, there’s one mental model to debug.

Strategy 3: isolate the fragile thing

The changelog generator lived inside a monolithic “heartbeat” workflow that also ran security scans, dashboard regeneration, and dependency enforcement. When the heartbeat’s build broke on those private dependencies, everything downstream of it — including the changelog — went dark. So we gave the changelog its own small workflow that installs only what it needs and runs in isolation. Now a build failure in one corner can’t take out an unrelated feature. Coupling is the silent tax you pay later; decoupling is the refund.

Strategy 4: one good credential, not five fragile ones

A surprising amount of the breakage traced back to authentication sprawl: an expired personal token here, an org-scoped app token that couldn’t reach across to another org there, a deploy key that wasn’t wired into the job that needed it. The cleanup was to mint a single, correctly-scoped credential and place it deliberately where each system reads it. One token now powers the blog commits, the workflow dispatch, and the cross-org changelog. Fewer moving parts, fewer things to silently expire.

The pitfalls, named

  • Silent drift. Autonomous systems fail quietly. If nothing is watching the watchers, “runs itself” becomes “rusts itself.”
  • Mirror-over-source. A force-push mirror will erase anything written directly to the downstream copy. Know which repo is the source before you write to it.
  • Format drift. A config file changed shape and a hand-rolled parser kept “working” — returning garbage instead of failing loudly. Parsers should reject what they don’t understand, not invent it.
  • Errors that mask errors. A lockfile error hid a dependency-auth error. Fix the loud one and brace for the quiet one underneath.
  • Credentials with quiet expiry. A 401 three months after the fact is the ghost of a token nobody renewed.

What it actually took

Not heroics — layering. Prove the architecture with a real experiment instead of arguing from memory. Move heavy work to a runtime that can finish it. Decouple the fragile pipeline so its failures stay local. Consolidate the credentials. And then, patiently, peel each error off the top and trust that the stack underneath is getting closer to correct with every layer removed.

By the end of the day the blog could publish again, end to end. The changelog was flowing real PR events back onto the timeline, autonomously, on a schedule. And the broader automation — security sweeps, dependency enforcement, the dashboard — was green for the first time in months. The workhorse is back in the harness, pointed down the right road. The trick was never one big fix. It was refusing to stop at the first one.