Turning Around the Workhorse

git-steer is a fleet-management workhorse — an MCP server that scans, fixes, and governs security across a few dozen GitHub repositories. It looked healthy: workflows fired, PRs appeared, dashboards rendered. The dangerous kind of healthy. Underneath, it was quietly lying about the fleet’s security posture and auto-merging things it had never actually verified.

This is the story of one day spent turning it around — not by rewriting it, but by establishing doctrine, building verification, and then root-causing a genuinely impressive cascade of bugs, each one hiding the next.

Part 1 — Doctrine before code

The day didn’t start with code. It started with a contract.

The first instinct when “fix the vulnerabilities” lands is to start bumping versions. We did the opposite: we wrote down what done means, as Architecture Decision Records, before touching the machinery.

  • ADR-004 — CVE Remediation Doctrine. Every CVE, every severity, reaches a terminal state: FIXED or SUPPRESSED-via-VEX. An SBOM for every repo. OpenVEX records when no fix exists. No “open and ignored.”
  • ADR-005 — Functional Integrity Verification. A remediation isn’t done when the alert closes — it’s done when the alert closes and the repo still works. merged ≠ working. This introduced a three-state pre-merge gate: GO / NO-GO / GO-UNVERIFIED.
  • ADR-006 — Autonomous Control Loop (drafted, parked). The gate feeds a self-driving escalation ladder instead of a human inbox: a NO-GO advances to a safer candidate fix; a human is the last resort, reached autonomously.

The third state — GO-UNVERIFIED — is the quiet hero of the whole day. It exists to defeat the false-GO trap: a green check on a repo that has no tests is not a pass. A two-state pass/fail collapses “verified working” and “nothing was checked” into the same green light. Naming the absence of verification as its own state is what made the autonomy safe.

Part 2 — The scanner was lying

The first concrete discovery set the tone. The canonical scan function queried Dependabot only and fetched one unpaginated page. For git-steer itself, it reported 2 of 23 real findings — a 9% view of reality — and silently dropped every code-scanning alert plus anything past the 100-item cap.

The fix was small (paginate both alert classes). The lesson was not: a truncated scan doesn’t fail loudly — it hands you false confidence. When we finally paginated the fleet scan, commit-relay jumped from a reassuring “100+” to its true 238 open findings. You cannot remediate what your tools are too polite to show you.

We validated every fix with self-contained harnesses (verify-app-scan, verify-sbom, verify-vex, verify-verdict) that exercise the real compiled code, not a copy. Each subsystem ships with a script that proves it on the actual artifact. That discipline paid for itself many times over.

Part 3 — Dogfooding: git-steer fixes itself first

Before pointing the machine at the fleet, we ran git-steer through its own doctrine end-to-end — the proving ground principle.

  • SBOM went from a hand-rolled JSON blob of 26 direct deps to a proper CycloneDX document of 223 components, sourced from GitHub’s full dependency graph (transitive included), pinned to a source SHA.
  • 17 CodeQL findings fixed in code (a global rate limiter, least-privilege workflow permissions, a pinned action).
  • 4 findings given honest OpenVEX dispositions — not_affected with specific justifications (the clear-text-logging finding already redacts tokens; the “user-controlled bypass” cases are operator config, not adversary input) — and the matching alerts dismissed atomically.
  • 2 transitive deps patched via npm overrides + a lockfile regen.

git-steer went from 23 findings to 0. The repo that manages the fleet’s security finally had clean security of its own.

We also built an append-only vex.jsonl ledger so VEX history stops being “diff a noisy cache file in git.” The very first time we ran the lookup tool, it surfaced four legacy malformed VEX entries — affected with no justification, written by an older script. The tool found its own predecessor’s mess on day one.

Part 4 — The continuity of errors

Here is the part worth remembering. Turning this workhorse around wasn’t one bug — it was a chain of them, each layer masking the one beneath it. You fix one, the next reveals itself. A field guide:

The worker had three bugs stacked on top of each other

The security-fix-worker — the thing meant to autonomously fix dependencies — failed across the entire fleet. Peeling it back:

  1. Truncation (the pagination bug above) meant it never saw most alerts.
  2. Fix that, and the fix step crashed under bash -e: a trailing wait "$NPM_PID" propagated a best-effort subshell’s non-zero exit and killed the whole step. set +e.
  3. Fix that, and it still failed — because when an ecosystem was absent, the last line [ -n "$GOMOD_PID" ] && wait … evaluated false, so the step’s final exit code was 1. exit 0.

Three independent failures, in one step, discovered one at a time. Only after all three did a fix PR actually get created.

lock-regen was doubly broken

The workflow meant to fix transitive deps couldn’t even be dispatched:

  1. Its gh pr create --body "…" heredoc had continuation lines at column 0, which dedent out of the YAML run: | block scalar and make the entire workflow unparseable. GitHub registered the file but couldn’t read its name — the tell-tale sign. (gh workflow view "Lock File Regeneration" → “could not find any workflows named…”)
  2. Fix the heredoc, and dispatch failed at Set up job: the repo enforces SHA-pinned actions, and lock-regen still used @v1/@v4/@v5.

Two more layers. Once both were fixed, it ran, self-merged, and cleared git-steer’s last two transitive CVEs.

”Removed” didn’t mean removed

We were asked to remove Snyk. The Snyk GitHub App had been removed long ago — yet security/snyk checks kept appearing on every PR. The culprit: two orphaned repo webhooks (api.snyk.io/webhook/…) that survived the app and kept firing events to Snyk, which posted statuses back via a stored token (giveaway: creator: null, not an app check-run). Deleting the app never deletes its webhooks. Always check repos/{r}/hooks.

CodeQL “failing in 3 seconds” was real

It looked like a stale/misconfigured required check — fails in 3s while the real Analyze jobs pass. It wasn’t. The 3s “check” is the diff-gate, and it was correctly reporting new high-severity alerts in the PR. The real lesson hid one layer deeper: CodeQL’s js/missing-rate-limiting does not recognize a global @fastify/rate-limit plugin — it flags every route handler regardless. The runtime mitigation is real, so the honest disposition is VEX not_affected / inline_mitigations_already_exist + dismiss — not “the plugin fixed it.”

The merge itself fought back

--admin bypasses required checks but not conversation resolution. Sixteen unresolved threads blocked the squash merge — all from CodeRabbit. Unlike GHAS threads, CodeRabbit threads are resolvable via the GraphQL resolveReviewThread mutation, so we cleared all sixteen programmatically and merged.

And zsh, the gift that kept giving

Three separate times, a for x in $LIST loop quietly did the wrong thing — zsh doesn’t word-split unquoted parameter expansions like bash. Once it fired a single dispatch with eleven repo names concatenated into one bogus argument. while IFS= read -r became muscle memory by mid-afternoon.

Part 5 — The strategies that actually worked

Threaded through the firefight were a handful of principles that held up:

  • Doctrine first. Settling the contract (ADR-004/005) before the code meant every fix had a definition of “correct” to satisfy, and the three-state verdict existed before we needed it.
  • Verify against the real artifact. Harnesses that import the compiled code caught drift a hand-wave would have missed.
  • Dogfood the hard case. git-steer remediating itself first surfaced the transitive-dep problem, the CodeQL-plugin blind spot, and the SBOM gap before they could bite the fleet.
  • Gate before merge, and name the absence of proof. GO auto-merges; NO-GO and GO-UNVERIFIED are held. The gate’s caution is a feature, not friction.
  • Honest suppression. Every VEX is not_affected with a specific justification or under_investigation (tracked) — never a blanket “won’t fix.” Each suppression is logged to an append-only ledger and re-evaluated on the next scan.

Part 6 — What autonomy actually achieved

With the workhorse running again, we pointed it at the fleet — 434 open findings across 13 repos. The gated worker, doing real work with no human in the loop:

  • Cleared 4 repos to zero (unifi-base44 56→0, proxasaurus, two MCP servers) — the gate gave GO because their builds passed, and the fixes auto-merged. unifi-base44 alone: 30 vulnerabilities patched and merged untouched by human hands.
  • VEX’d 47 no-fix deps (torch/mlflow/nltk — ML packages with no upstream patch) to a tracked terminal state.
  • Held 4 PRs with honest verdicts: three NO-GO (the batch fix broke the build — correctly not merged) and one GO-UNVERIFIED (no tests to verify).

That last line is the whole point of the gate. It refused to auto-merge fixes that broke the build or that nothing had verified — exactly the blind merges that the old “merged = done” worldview would have shipped.

It also drew the honest boundary of fully-autonomous work. Repos whose vulns are transitive with no regenerable lockfile resist both tools. Real SAST findings (an SSRF in one app) can’t be blind-fixed or blind-dismissed. Those need the escalation ladder (ADR-006) or a human — and saying so is part of doing it right.

The takeaways

If there’s a thesis under all of this, it’s three sentences a tired machine kept trying to teach us:

  1. Green is not working. A passing check on an untested repo is the absence of information, not the presence of safety. Give it its own name.
  2. Merged is not fixed. Closing an alert and leaving a working repo are two different events. Verify the second one before you celebrate the first.
  3. Removed is not removed. Apps leave webhooks. Scanners truncate. Plugins go unrecognized. The thing you “handled” has a habit of one more layer.

Turning around a workhorse this size wasn’t a heroic rewrite. It was doctrine to define correctness, verification to enforce it, and the patience to root-cause a cascade of bugs where every fix uncovered the next. The machine is back on the rails — scanning the whole truth, fixing what it can prove, and honestly holding what it can’t.

— a day’s work, 2026-06-14