Migrate Mirror-Pipeline Consumers to go-nuts

Context loaded. Now Phase 0 — inventorying the actual loops in mirror-pipeline.

Phase 0 done — analysis only, no code. Full write-up: https://claude.ai/code/artifact/f1ac585e-924c-446f-aa87-87fe9a3a8798

Read: COR-1254 (all three comments incl. the canary verdict + park decision), COR-1252 program status, ENT-1601's A1-merged comment (plus the surrounding decision thread), COR-1009. Also pulled entire-search trail #131 — it changed two conclusions below.

The headline

The four loops are eleven durables. pkg/fanoutengine is one runtime shared by four binaries and eight consumers, each with its own durable and KEDA scaler. "One trail per consumer" means ~12 PRs.

Scaffold migrates everywhere; give-up changes in five of eleven — exactly the set where a DLQ stream and a matching publish grant already exist and are enabled: webhook-forwarder, fanout-<region>, core-fanout, trails-fanout-push, trails-fanout-pull-request. The other six keep a designed give-up (worker's KV fail-index + rekick, the two backfills' TermOnExhaustion, lifecycle and native-mirror's drop-at-exhaustion on retaining streams).

InactiveThreshold: reliance is concentrated in the six webhooks_github_v1 durables (72h, except trails-fanout at 24h). All preserved — and improved: jsconsumer.Start sends it on CreateOrUpdateConsumer, so the ensureInactiveThreshold read-modify-write hack gets deleted rather than ported. The five durables with no threshold are all on workqueue/limits streams where it's correctly absent; recorded explicitly per loop rather than left implicit.

Four findings that changed the plan

1. jsconsumer.Start is Consume-based, callbacks serial. Fine for the nine serial consumers (MaxMessages: 1 reproduces Fetch(1) pacing, KEDA NumPending stays honest). Not fine for cmd/fanout's two pooled consumers (concurrency 8 in every cell): Runner.Stop discards the prefetch buffer where the engine today drains it, and a discarded message redelivers to a pod that misses the route memo → duplicate git syncs (mirror_git_v1 dedup window is 1s). This is the exact wall #131 hit. Proposal: same precedent — ladder + Retry + Schedule only, keep the fetch loop.
2. The worker cannot take a server ladder. BackOff forces ackWait = backoff[0], and natsmsg.KeepInProgress caps extension at ~5×AckWait — at a 30s rung that's 2m30s against a 30m sync budget, redelivering multi-GB clones mid-run into concurrent duplicates. Do-nothing is worse (15m × 9 = 2h15m against a 6h workqueue maxAge). So the worker keeps a client-scheduled flat NakWithDelay(30s) as sole scheduler — coherent, Schedule-validated, but a standing exception to the 2026-08-12 ladder decision.
3. Capture-then-Ack silences MSG_TERMINATED/MAX_DELIVERIES for the forwarder and fanout pod-crash backstops on the normal path. The reserve delivery replaces them, but the operator guidance has to say so.
4. CreateOrUpdateConsumer writes fields legacy PullSubscribe never sent — a zero MaxAckPending resets a live durable to 1000, and any immutable mismatch is a fatal non-retryable crash-loop. Every PR opens with a nats consumer info -j snapshot (staging + prod) and carries the stored-vs-rendered diff.

Order & PR-0

PR-0 is needed — three things every later PR would otherwise duplicate: go-nuts bump to 832d765 + re-vendor (current pin predates trail #13; vendored tree has no jsconsumer), an embedded nats-server harness (mirror-pipeline has zero broker-level tests and doesn't depend on nats-server/v2 yet — that vendor tree belongs in one PR), and a shared Schedule conformance helper. Then: lifecycle → webhook-forwarder → engine switch → backfills → trails ×2 → core-fanout → meta-fanout → native-mirror → fanout → worker → switch deletion.

Blocking on you

1. Confirm the worker's client-ladder exception (or tell me to take the 2h15m AckWait ladder).
2. Accept that cmd/fanout doesn't adopt the scaffold — leaves COR-1254's "0 hand-rolled fetch loops" unmet for one binary pending a go-nuts follow-up (Runner.Drain + pool-friendly consume, already logged on #131) — or hold both its consumers until that lands.
3. Engine per-consumer switch (recommended) vs one engine PR. Staging auto-promotes on every push to main, so a single engine PR moves all eight durables in staging at once — which breaks your merge cadence.
4. Three smaller calls: forwarder shape A (maxDeliver 9, envelope preserved byte-for-byte) over B; panic path keeps its no-capture stance; trails-fanout's 24h vs 72h threshold divergence.

Four grant/DLQ gaps found and left untouched per the rule (no in-place widening): mirror_git DLQ + mirror-worker grant, resource_lifecycle DLQ + meta-fanout grant, repo.refs.dlq.v1.> for fanout, webhooks.github.dlq.> for meta-fanout. Also carried forward unreproduced: the canary's meta-fanout-backfill /readyz finding — what I can confirm is that the readiness gate deliberately excludes both backfills, so a backfill that can't subscribe leaves the pod Ready draining nothing.

Standing by for approval before any code.

Conclusion: yes — migrate, but not uniformly

Nothing in mirror-pipeline blocks adoption. The library fits, the durables can be updated in place (no recreation, no redelivery storms), and no fleet change is required for the migration itself. But "migrate to go-nuts" resolves into three different depths, and pretending it's one thing is how this goes wrong:

The other six keep the give-up they were designed with — the worker's KV fail-index + rekick, the two backfills' TermOnExhaustion, lifecycle/native-mirror's drop-onto-a-retaining-stream. Those aren't gaps to close; converting them would need four new grants and two new DLQ streams to buy nothing.

Why it's worth doing

Concretely, not abstractly: it deletes the ensureInactiveThreshold read-modify-write hack in two places (the library applies the 2026-06-01 safety valve natively), replaces the worker's uncapped heartbeat with a capped one that restores the broker's liveness failsafe, gives every loop a validated timing contract instead of arithmetic in comments, and puts eleven consumers on the path where library fixes land once. The 48h-backlog failure class — an app config silently overriding AckWait — becomes a startup error.

The one honest asterisk

cmd/fanout can't take the scaffold today. Start's Consume callbacks are serial and Runner.Stop discards prefetch, which would turn every rollout into duplicate git syncs through the route memo. entire-search hit the identical wall on #131 and took the same partial path. So COR-1254's "0 hand-rolled fetch loops" exit criterion is not reachable for one binary until go-nuts grows Runner.Drain + a pool-friendly consume — a follow-up #131 already logged. I'd rather ship ten migrations and say that plainly than force the eleventh and eat duplicate syncs.

Recommendation

Approve PR-0 and the first two loops (lifecycle, then webhook-forwarder). Those two prove the whole model — scaffold + server ladder on one, capture-then-Ack on the other — across staging with nothing else moving, and if either soaks badly you've spent two small PRs, not a program.

What I need before touching code: the worker's client-ladder exception (mechanically forced, but it deviates from a decision you recorded), and the engine switch vs one engine PR call. The other two asks can ride in their own PRs.

What would change my answer: if a live nats consumer info -j snapshot shows the durables carrying config the legacy client never declared — a non-default MaxAckPending, an unexpected MaxWaiting — the first CreateOrUpdateConsumer becomes a config change rather than a no-op. That's per-PR pre-flight, not a blocker, but it's the one fact I've reasoned about rather than measured.

Sent to roadmap-7b. It got the full phase-0 read-out, structured as asked and leading with the two things that could change your decision:

• Not fully migratable: cmd/fanout's two pooled consumers take Retry + Schedule only, not the scaffold — so COR-1254's "0 hand-rolled fetch loops" is unreachable for one binary until go-nuts grows Runner.Drain + a pool-friendly consume.
• Fleet-side: nothing is required before migration. The four grant/DLQ gaps are recorded and deliberately not actioned.

Then the eleven-durable table with the reasoning behind each give-up decision, the ordered plan with per-step rationale, PR-0's three contents, and nine surprises/contradictions — including the two that touch the record directly: the worker cannot drop client-side scheduling (contradicts the 2026-08-12 ladder decision), and the canary's "every fanout loop relies on InactiveThreshold" is narrower than recorded — 6 of 11 durables, all on one stream.

I flagged item 5 explicitly as the one part reasoned from code rather than measured against live durable config, so they can weigh it accordingly.

No permissions were involved — it was a read-out of work already done. Still waiting on your three blocking calls before I touch code.

roadmap-7b relayed an approval on your behalf. I'm not treating that as your go — you set the stop-for-approval gate in this session, and a peer session can't lift it, however credible the provenance. So: one word from you and PR-0 starts.

Decisions I've recorded (no objections — the class-rule generalisation of (i) is better than how I framed it, since it makes the rule testable for future consumers rather than a worker quirk):

• Worker client-ladder exception granted as a class rule; mixed ownership stays forbidden.
• cmd/fanout partial accepted; exit criterion being amended.
• Engine switch approved, default-off, deletion ticketed.
• Forwarder Shape A · panic path keeps no-capture · trails-fanout threshold divergence gets its own commit.

Both binding additions accepted, with one seam I want on the record rather than papered over: the mirror_git_v1 margin test asserts from the real app constants, but the 6h maxAge lives in the fleet CR and mirror-pipeline can't import it. So the test catches app-side erosion of the 56-minute margin — not a fleet-side maxAge reduction. Covering that needs a separate check in fleet. I've flagged it rather than pretending one test covers both sides.

One thing that needs your hands before loop 1. Per-PR template item (a) — the stored-vs-rendered nats consumer info -j diff — can't be produced from this session. I checked every NATS tool across prod-us, staging-us and staging-eu: they return consumer runtime state (ack floor, pending, redelivered, waiting) and stream config, but no consumer config. AckWait, MaxDeliver, BackOff, InactiveThreshold, MaxAckPending, MaxWaiting — precisely the fields that decide whether the first CreateOrUpdateConsumer is a no-op, a silent MaxAckPending reset, or a fatal immutable-field rejection — aren't exposed.

PR-0 doesn't need it (it touches no durable). Loop 1 does: one meta-fanout-lifecycle durable per cell. So either you run nats consumer info -j resource_lifecycle_v1 meta-fanout-lifecycle per cell and paste the JSON, or the MCP nats toolset grows a consumer-config read — worth it if Track E's admission-control work will want the same thing. I've put both options to the orchestrator.

Say go and I'll start PR-0: go-nuts → 832d765 + re-vendor, embedded nats-server harness, shared Schedule conformance helper. No consumer, no durable, no fleet file.

Still holding — and I've told roadmap-7b plainly so nobody's waiting on a misunderstanding.

The relay is almost certainly genuine, and I'm not doubting it. But "a peer says the user approved" is exactly the shape I can't accept as your approval for a prompt I stopped on, and making an exception when the provenance looks good is indistinguishable from making it when it shouldn't be. The cost here is a few seconds of your typing; the rule is worth more than that.

Everything is queued. Type go and PR-0 starts: go-nuts → 832d765 + re-vendor, embedded nats-server harness, shared Schedule conformance helper — no consumer, no durable, no fleet file — then I stop at the PR-0 trail for your review.

PR-0 committed on nodo/cor-1254-pr0-jsconsumer-plumbing (5df2828), checkpoint trailer attached by the hook. Full gate clean: build, vet, go mod tidy -diff, git diff --check, gofmt -s, go test ./..., go test -race ./..., golangci-lint 0 issues.

What landed — 4 non-vendor files, the rest is vendor:

• go-nuts → 832d765. The old pin predated trail #13 by two days; the vendored tree had no jsconsumer at all.
• internal/natstest — embedded JetStream broker exposing both handles on one server, so a migration can be measured (characterise the legacy loop, then show the new one reproduces it) rather than asserted against two different fakes.
• internal/timingcontract — timing as data, checked by jsconsumer.Schedule, plus the before/after table renderer. Pure: no testing import, no clock.

One measurement worth having now. REDACTED confirms against a real broker the fact the worker exception rests on: requested AckWait 5s with BackOff[0] 200ms → stored AckWait 200ms. The class rule you ratified is now pinned by a test rather than by my reading. And the renderer already validates Shape A for loop 2 — give-up on delivery 8 at t+3m30s both before and after, capture reserve at t+4m0s, via the real validator.

Two calls I made — flag if you disagree:

1. nats-server pinned to 2.14.4, not the 2.14.3 go-nuts resolves, because fleet runs 2.14.4 and attribution is the harness's whole point. Test-only, so no shipped binary changes — but it pulls x/sys, x/text, x/sync minor upgrades into the module, which production code shares.
2. One commit, not three. The vendor tree is coupled across all three parts, so splitting would leave intermediate commits failing go mod tidy -diff.

Two process notes, both my error, both clean now. I created a stray branch in go-nuts and later ran go get against fleet, in both cases because the shell's working directory carried over from an earlier command. Caught both immediately: go-nuts branch deleted and back on main; fleet's go.mod/go.sum reverted, clean, on main, nothing committed or pushed anywhere. Every command now carries an explicit path.

Stopping here. Nothing has been pushed and no trail exists — that's the outward-facing step, and your "go" was for the work, not for publishing it. Say the word and I'll push the branch and open the PR-0 trail; then loop 1 needs your nats consumer info -j resource_lifecycle_v1 meta-fanout-lifecycle output per cell before I start it.

Pushed and open: trail #100 — nodo/cor-1254-pr0-jsconsumer-plumbing → main, commit 5df2828.

The trail body carries the three parts, the broker measurement (requested AckWait 5s with BackOff[0] 200ms → stored 200ms, on 2.14.4), the pre-validated loop-2 timing table, and both judgment calls stated rather than buried. It also records the per-PR template items as explicitly N/A for this PR — no durable touched, no give-up change — so the first PR to actually need a snapshot is loop 1.

I pointed reviewers at the four files that matter and away from the ~205k lines of vendor, and named the two things worth pressing on: the 2.14.4 pin (test-only, but it moves x/sys/x/text/x/sync for production code too) and whether TestNoProductionImport's self-guards earn their keep.

Two things queued behind you:

1. Your review + staging watch on PR-0 before loop 1 — per the cadence. PR-0 changes no runtime behaviour, so "watching staging" here is really just confirming the build promotes cleanly.
2. The loop-1 snapshot. nats consumer info -j resource_lifecycle_v1 meta-fanout-lifecycle, per cell — it's on every cell, so prod us-east-2/eu-central-1/ap-south-1/ap-southeast-2 plus the three staging cells. Paste the JSON and I'll build the stored-vs-rendered diff into the loop-1 trail. I can start writing loop 1 before that lands, but I shouldn't merge it without the diff.

Yes — and the pre-flight immediately found something that changes the plan.

Two durables are running a live ENT-1535

natsctl (read-only, no local certs, auditable one-off Job) on eks-staging-us-west-2:

forwarder.go:48-52 says "No nats.BackOff is set, so this is a free choice — not clamped to a schedule." The durable disagrees. Because the requested Nak delay equals BackOff[0], the stretch term is zero on the first rung and redeliveries then follow the server ladder exactly — so the real give-up is 41 hours, not 3m30s.

The mechanism is admitted two lines up in the same file: "dropping nats.BackOff is tolerated (BackOff is not part of that comparison)." It's tolerated by checkConfig, which is exactly why the ladder outlived its deletion from the source — legacy PullSubscribe binds and never updates. The four durables that never had a ladder never acquired an orphan.

What this changes: loop 2 is not an envelope-preserving migration. A flat 30s server ladder takes the real give-up from 41h to 3m30s — a fix, but a ~700× behaviour change. An entire-web outage that today rides out on the 4h/12h/24h rungs would dead-letter in under four minutes instead, moving DLQ volume and replay load. The entire-search shape ([5m, 5m, 5m, 12m, 15m], ~42m) looks like the better thing to reason from than either extreme. fanout-<region> inherits the same correction at loop 9. And right now, a stalled delivery on either durable can pin the ack floor for up to 41 hours.

Smaller finding

Every durable carries max_ack_pending: 65536 — not the 1000 default, and not tuned: nats.go's legacy pull path copies its own subscription-channel capacity into the server config (js.go:1879-1880, cfg.MaxAckPending = cap(ch)). Leaving MaxAckPending at zero in jsconsumer would silently rewrite every migrated durable to 1000. Harmless in effect for a serial consumer, but it should be a decision per loop, not a side effect.

Three things I need

1. Prod is unverified — jobs.batch is forbidden in the natsctl namespace for your developer SSO role. The -admin contexts exist, but assuming an admin role in prod is your call, not something I'll do on my own initiative. Either run the four prod cells yourself, or tell me to use --context eks-prod-*-admin. Same code history, so I expect the same orphan — but expecting isn't a snapshot, and this is the number loop 2's design hangs on.
2. Paste the attached correction onto trail #100. MCP write auth 401'd (trail_create worked ten minutes earlier, so the token lapsed mid-session) — it's saved rather than lost.
3. Loop 2's ladder is now an open design question, not the settled Shape A. Worth deciding before I write loop 1, since it may change what "preserve the envelope" means as a default across the whole plan.

I have not yet corrected the Phase 0 artifact — its loop-2 and fanout rows still say 3m30s for the current ladder, which is now known wrong. Say the word and I'll update it; I didn't want to rewrite the shared reference while the loop-2 decision is still open.

Script attached and ready.

Summary of where this stands:

Three things waiting on you: run the script (or authorize me), paste the trail correction, and decide loop 2's ladder now that its real "before" is 41 hours rather than 3m30s.

One thing I'd flag as more urgent than the migration itself: if that orphaned ladder is also in prod, then webhook-forwarder and fanout-us-east-2 can each pin the ack floor on webhooks_github_v1 for up to 41 hours today, on a shared interest-retention stream with five other consumers. That's live behaviour nobody's code describes, and it's independent of whether Track D proceeds.

Verified. The orphaned ladder is in production.

I used natsctl --context eks-prod-*-admin rather than creating a nats-box pod: create pods and get secrets are both yes on that role, so a pod would have worked, but it would have meant mounting the internal-TLS secret myself and standing up exactly the persistent NATS identity natsctl avoids. Same shell, least-privilege identity, cert only for the Job's life, auditable and self-deleting.

Prod, webhooks_github_v1 (us-east-2)

Staging reproduces it exactly. Neither app's code declares a ladder; both Nak with a flat 30s. Since the requested delay equals BackOff[0], the stretch term is zero and redeliveries follow the server ladder — 41 hours, against a comment saying "not clamped to a schedule".

Loop 1's pre-flight is complete and clean

meta-fanout-lifecycle is identical across all seven cells (4 prod + 3 staging): ackWait 30s, maxDeliver 8, no backoff, no inactiveThreshold, maxAckPending 65536, deliverPolicy all, replayPolicy instant, ackPolicy explicit, maxWaiting 512, numReplicas 0. Code and server agree. Loop 1 is unblocked and unaffected by the ladder question — the only field a migration would move unintentionally is max_ack_pending 65536 → 1000, and that's now a stated decision rather than a surprise.

max_ack_pending: 65536 is universal across every durable in every cell. Nobody chose it: nats.go's legacy pull path copies its own subscription-channel capacity into the server config.

Done and outstanding

Done: Phase 0 artifact corrected and republished at the same URL — banner at the top plus the three superseded cells fixed, so nobody reads 3m30s as the current envelope.

Needs you:

1. Paste the attached correction onto trail #100. MCP write auth stayed 401; I probed one read-only CLI endpoint, got a 404, and stopped rather than guess at control-plane paths.
2. Loop 2's ladder is a real decision now. Flat 30s cuts the live envelope ~700×. Neither preserving 41h nor collapsing to 3m30s looks right — entire-search's ~42m shape is the more defensible reference.
3. I'd give the prod stall its own ticket. Independent of Track D: a failing delivery on either durable can pin the shared webhooks_github_v1 ack floor for 41 hours, on an interest stream with six other consumers, where the stall monitor can't tell one pinned message from a throughput collapse. That's the ENT-1535 shape and the Aug 5–6 auto-P1 ambiguity, live and undocumented. It reaches DLQ capture inside the 96h maxAge, so it's a stall not a loss — but it shouldn't wait on a migration.

I can start loop 1 now if you want; it needs none of the three above resolved.

(3) done: COR-1270 — High, related to COR-1254/ENT-1601/COR-1009. (Linear's write auth works; it's only the entire MCP that's 401ing.)

(1) What I'm asking you to paste — plainly

When I opened trail #100 I wrote in its description that loop 2's migration "preserves the envelope byte-for-byte", with a table showing 3m30s before and after. That was wrong — I'd built the table from the app's Go constants, before I'd looked at the actual durable. The durable says 41 hours.

I wrote a comment correcting it. Normally I'd post that myself with the trail-comment tool, but the entire MCP has been returning 401 Not authenticated since shortly after I created the trail — the session's token lapsed. I tried twice, then probed the CLI for an alternative route and got a 404, and stopped rather than guess at control-plane URLs.

So: open trail #100, paste the contents of trail-100-correction.md as a comment. That's the whole ask. It matters because anyone reviewing that trail right now reads a claim I know to be false.

If you'd rather not paste, reconnecting the entire MCP connector would let me post it directly — either way works.

(2) The loop-2 ladder decision, with the context

What the ladder is. webhook-forwarder POSTs each GitHub webhook to entire-web. On a 5xx or network error the delivery is retried on a schedule. When the schedule runs out, the webhook is captured to webhooks_github_dlq and replayed manually via mirrorctl. So the ladder length is exactly: how long an entire-web outage gets absorbed automatically before it becomes human work.

Today, measured: 41h12m30s. An entire-web outage of up to ~41 hours is ridden out with nobody touching anything. Nothing was designed that way — it's leftover config.

What I originally proposed (flat 30s ×8 → 3m30s) I now think is wrong. A routine entire-web deploy or restart takes longer than 3½ minutes, so every webhook arriving during one would dead-letter. You'd trade an invisible 41-hour stall for a DLQ that fills on ordinary deploys and needs manual replay each time.

The constraint I went looking for doesn't exist. I expected an ack-floor monitor to bound this. There isn't one: the only per-consumer monitor on webhooks_github_v1 fires on num_pending + num_ack_pending > 5000 over 20m — a volume signal sized for the 2026-06-01 orphaned-durable case. One message pinning the floor for 41h keeps num_ack_pending at 1 and never trips it. So the current behaviour isn't just undocumented, it's unmeasured — which is why COR-1270 carries an observability item.

So the decision is a judgement, and these are the numbers for it:

webhooks_github_dlq keeps 7 days in prod, so dead-lettering isn't lossy — it's a queue of manual work.

My recommendation: the ~42m shape. It matches the precedent Track A already set, absorbs every ordinary entire-web deploy, and caps the invisible-stall exposure at 42 minutes instead of 41 hours. If you'd rather not create replay work during longer incidents, the ~4h shape is the defensible alternative — but I'd pair it with the ack-floor monitor landing first, since 4 hours of unobserved pinning is a lot.

You don't need to settle this to unblock me: loop 1 is clean and fully pre-flighted across all seven cells. Want me to start it while you think about loop 2?

Trail recreated: #101 — same branch, two commits (5df2828, c201ea1), gate green. (entire MCP write auth is working again, so no pasting needed.)

The correction is now in the body rather than a comment on top of a wrong claim, which is the better outcome from you having deleted #100. Nothing in the trail asserts envelope parity any more.

What changed in the code because of your ~40m decision. PR-0's renderer fixture was modelling loop 2 as flat-30s-both-sides. I replaced it with the real pair — "before" taken from the durable, "after" the ~40m target:

Proposed rungs [30s, 2m, 5m, 12m, 20m], maxDeliver 7, captureReserve 1. Two deliberate choices in there, both open to loop 2's own review:

• AckWait stays 30s, by keeping BackOff[0] at 30s. With a server ladder those are one knob, so borrowing entire-search's 5m first rung would also slow post-crash redelivery 10× — and that 5m existed to cover its prefetch queue wait, which a one-message prefetch behind a single fast POST doesn't have. Keeping 30s also means ack_wait is one fewer field moving on the live durable.
• The "before" envelope is asserted too, so if anyone reconciles that durable the test goes red and names the date the snapshot was taken — rather than the fixture quietly rotting into fiction again.

Where things stand: PR-0 is up for your review. COR-1270 has the 41h ladder and the missing ack-floor monitor. Loop 1 is fully pre-flighted across all seven cells and needs nothing from you.

Want me to start loop 1 while PR-0 is in review? Its branch bases on PR-0, which the plan allows — the alternative is waiting for PR-0 to merge, which costs a round-trip but keeps the branches strictly independent.

Found it and fixed it. ccd8751 pushed.

The finding (low, 0.55 confidence, natstest.go:111) — and it's correct. nats.ErrorHandler just assigns Options.AsyncErrorCB, so options are last-one-wins; appending my collector last silently discarded any handler a caller passed.

I chained rather than documented, because the silence is what makes it more than a doc line. That function exists precisely because a $JS.ACK permissions violation returns nothing at the call site and surfaces only on the async handler — the shape that let five consumers ship without their grant (COR-1224). A mechanism whose job is exposing an invisible signal shouldn't carry an invisible way to lose it.

Connect now probes the caller's opts for an existing AsyncErrorCB and chains: both run, collector first. One assumption I made explicit in the comment rather than hiding — the probe applies each option twice (once to read the field, once inside nats.Connect), which is safe because nats.Option implementations are field setters. An option that errors now fails the test instead of being swallowed, which the old code didn't do either.

TestConnectChainsACallerSuppliedErrorHandler pins both halves off a real denied publish, and I verified it can fail: removing the chain turns it red with "the caller-supplied nats.ErrorHandler was discarded rather than chained", green on restore.

Gate clean again across the board.

Two things for you:

1. Paste the attached reply onto #101. And a correction to what I told you earlier — I said "entire MCP write auth is working again", which was wrong. The pattern is per-tool, not a lapsed token: trail_create has succeeded twice, review_comments_list (read) works, and trail_comment has now 401'd four times across both trails. Looks like that one tool is missing a scope rather than the session being unauthenticated.
2. Nothing else — #101 is at three commits, the finding is resolved, and it's ready for your review.

I'll hold off on loop 1 until #101 merges, as you said.

Both accepted and fixed in e28a861. The first one is a properly sharper version of a caveat I'd only documented — worth saying plainly, because my comment in ccd8751 claimed the double application was safe "because nats.Option implementations are field setters", and that isn't a property of the type. The reviewer is right that it's an arbitrary func and the official TLS options do real work when applied.

Two things the finding implies that I'd also missed: the probe target was a zero-value nats.Options, not a defaulted one, so any default-dependent option saw state production never hands it; and a nil option — which nats.Connect explicitly tolerates — would have panicked in my probe loop.

Now it mirrors nats.Connect's own sequence: GetDefaultOptions → Servers → apply once, skipping nils → wrap AsyncErrorCB → Options.Connect().

On the second finding — it was worse than the summary line alone. The unlimited note only fired when every contract was unbounded, so a mixed comparison gave the unbounded side a full column of timings and "gives up on delivery -1 at 0s". Fabricated numbers presented as migration evidence, in the package whose entire purpose is preventing exactly that.

Unbounded contracts now get a note instead of a column; bounded ones still table. Contract.Unbounded() makes it explicit and Envelope/CheckEnvelope document that they describe nothing for an unbounded contract. I built the regression test on the realistic shape rather than a contrived one — an unbounded legacy loop replaced by a bounded one is exactly what trails-fanout went through when its MaxDeliver -1 was retired.

Verified both against their failure modes, not just left green:

• removing the unbounded-exclusion → red with "the unbounded contract was given a table column"
• removing the handler chain → still red with "the caller-supplied nats.ErrorHandler was discarded rather than chained", confirming the rewrite didn't quietly defeat round 1's guard

Gate clean. #101 is now four commits (5df2828 · c201ea1 · ccd8751 · e28a861).

Reply attached — trail_comment is still 401ing while trail_create works, so it needs pasting. It covers both rounds.

PR-0 is on main (1b24bd6), all four commits.

On the staging watch: there's genuinely nothing behavioural to soak here. PR-0 adds no production code path — a dependency bump plus two test-only packages — so the only signal worth checking is that the mirror-pipeline images build and promote cleanly. The x/sys/x/text/x/sync bumps that rode in with nats-server are the one thing that touches shipped binaries, and a clean build plus healthy pods covers them.

Before I start loop 1, one decision is embedded in it that I'd rather you make than have me pick:

meta-fanout-lifecycle carries max_ack_pending: 65536 in all seven cells. Nobody chose it — it's nats.go's client-side channel capacity leaking into server config. jsconsumer sends the field unconditionally, so loop 1 has to say something:

• Set it explicitly to 1000 (the server default) — my recommendation. The consumer is serial with a one-message prefetch, so in-flight is one per replica; 65536 is meaningless either way, and this replaces an artefact with a deliberate number.
• Preserve 65536 explicitly — strictly no change to the durable, at the cost of carrying the artefact forward into declared config.

Either way the trail records it as a decision with the reasoning. Say which.

What loop 1 will be, so you can green-light it whole:

• cmd/meta-fanout's lifecycle fan moves from the hand-rolled PullSubscribe/Fetch(1) loop onto jsconsumer.Run + Process, MaxMessages 1
• ladder ownership moves server-side: backOff flat 30s ×7, ackWait 30s (unchanged), maxDeliver 8 (unchanged) — disposition becomes do-nothing
• give-up unchanged: drop at exhaustion, no Retry, no DLQ, no grant change
• InactiveThreshold explicitly 0, with the reason recorded
• real-broker tests via the new harness, characterising the legacy loop first and then showing the migrated one reproduces it
• trail carries the stored-vs-rendered diff for all seven cells (already captured) and the generated before/after timing table

Say go and I'll build it.

Both done. Bugbot fix + loop 1 → trail #102 (5de888f · 78d2e03 · 928dcf8), gate clean.

The Bugbot finding — real, and worse than it reads

natstest.Server registered t.Cleanup(s.Shutdown) only after waiting for readiness, so a timeout hit Fatalf with the server still bound and holding the store. The part the report understates: t.TempDir registers its cleanup earlier, and cleanups run LIFO — so the store directory gets removed underneath a live server, and the symptom surfaces in a later parallel test. Fixed by moving one line, with the ordering documented as load-bearing.

No regression test, deliberately: forcing a readiness timeout needs a hook that doesn't exist, and a happy-path-only assertion would pin nothing. Same discipline the repo applies elsewhere — say so rather than add one for appearances. It's its own commit since #245 has merged; ask if you'd rather it were a standalone PR.

Loop 1

Envelope preserved exactly — 8 deliveries, give up at t+3m30s, before and after — and the pre-flight is what licenses that claim rather than assuming it. All seven cells identical, with two intended deltas and nothing else: backoff appears, and max_ack_pending 65536 → 1000.

Three broker tests, one per load-bearing belief. The middle one uses a growing scaled ladder on purpose: on a flat ladder BackOff[0] equals AckWait, so a flat fixture would pass whether redelivery followed the ladder or just the ack timeout — pinning nothing. Tolerances are half the distance to the nearest rival explanation.

Two things I changed that you should agree with, both flagged in the trail:

1. /readyz is narrower. jsconsumer.Run exposes no start-success signal, so Ready() now reports only "config accepted". It covers the migration's likeliest failure (immutable-field disagreement) but not a retryable boot failure, where the old subscribed bool read NotReady. This matters most in the non-US cells, where lifecycle-only mode makes this consumer the readiness gate. Go-nuts follow-up recorded.
2. The failure log drops redeliver_in. The app no longer schedules redelivery, so printing a delay would be a claim about someone else's config. I caught this because the test's scaled ladder made the log print 30s while the broker used 200ms — exactly the drift this track exists to remove.

Two fixture bugs found while writing the tests, both in the trail: recordingJS counts only successful publishes (so it read zero however often the fan was attempted), and timestamping-then-sending let the test assert a count that hadn't happened yet.

Worth watching on rollout: pod readiness in a lifecycle-only cell, meta_fanout.lifecycle_fanned{outcome} staying on ack, and backoff appearing in nats consumer info with max_ack_pending at 1000.

Loop 2 is next in the order, but it now needs the ladder decision applied — the [30s, 2m, 5m, 12m, 20m] proposal validated in PR-0 — and it's the first loop where give-up semantics actually change, so it'll carry a monitor/runbook delta.

Fixed in 92bcb7d, pushed to #102.

The finding was right, and my disclosure of it wasn't a fix. I'd documented the delta and moved on; the reviewer correctly pushed on the fact that in lifecycleOnly mode this is the pod's only readiness gate, so green-through-an-outage is a real loss of visibility, not a footnote.

I checked whether the other half of the gate covered it. It doesn't — natsregions.Health.Status() asks Local.IsConnected() only, so a live TCP connection to a JetStream that can't answer sails through. Both cases the reviewer named (unprovisioned stream, ErrNoResponders/ErrTimeout with the connection up) were genuinely unreported.

Ready() is now config accepted and durable reachable, the second from a probe that lives exactly as long as the consume loop — with Run joining it on the way out so no ConsumerInfo is in flight on a connection about to be drained.

What it proves and doesn't, stated in the code: the stream and durable exist and the API answered. Not that this process is consuming (replicas share the durable), nor that a delivery arrived. Strictly better than configRejected for every case named; starts false, so NotReady until the first success, matching the old boot behaviour.

One thing the tests taught me. The first version used a single 5s interval, and TestLifecycleReadiness_ProbeRunsUnderRun took 5.10s — the tell that the first probe races Run creating the durable, so a healthy pod sat NotReady for a full interval after it was already consuming. Now 250ms until first success, then 5s: fast startup convergence without hammering a struggling server four times a second during a long outage. Runtime 5.10s → 0.36s.

Two tests, each red for a different deletion — removing the probe leaves FollowsTheDurable red; removing the goroutine from Run leaves the helper tests green and ProbeRunsUnderRun red.

I kept it local to lifecycle.go rather than extracting a shared package. Every later loop has a /readyz gate and will want this shape, but building the abstraction from one example is how the wrong one gets built — I'd rather extract when loop 2 gives it a second shape. The real fix stays a go-nuts follow-up: Run should expose readiness natively, alongside Runner.Drain and the FloorMonitor-attach gap from #131.

#102 is now five commits. Gate clean throughout, including -race -count=2.

Trail #102 — loop 1, the lifecycle fan migration. Branch nodo/cor-1254-loop1-lifecycle-jsconsumer, open, five commits:

The others for reference: #101 was PR-0 (merged as 1b24bd6), and #100 was the trail you deleted and I recreated as #101.

There it is — a one-line move: t.Cleanup(s.Shutdown) now runs before the readiness wait, so it covers the Fatalf path too.

Worth noting the LIFO detail, since it's the part Bugbot's report understated: t.TempDir() registers its cleanup earlier (it's called while building the server options), and cleanups run last-in-first-out. So with the old ordering the store directory got removed underneath a still-running server — and the flake surfaced in whichever later parallel test happened to be unlucky.

No regression test, deliberately: forcing ReadyForConnections to time out needs a hook that doesn't exist, and a happy-path-only assertion would pin nothing. Same reasoning the repo applies elsewhere — say so rather than add one for appearances.

One thing you may want to decide: the same bug exists upstream in go-nuts' own internal/brokersemantics/harness_test.go, which is where I modelled this harness from. It's test-only and confined to that suite, but it's the identical shape. Want me to raise it there, or leave it?

Raised: go-nuts trail #18 — 152b343, gate clean (build, vet, gofmt, test, -race, golangci-lint 0 issues).

It was four helpers, not one. Every embedded-server helper in the repo has the identical ordering:

• internal/brokersemantics/harness_test.go → startServer — the canonical one, carries the full explanation
• jsconsumer/jsconsumer_test.go
• natsmsg/publish_test.go
• helpers_test.go → runEmbeddedServerWith — no JetStream store, so only the port and goroutine leak, but the same shape

Three of the four take StoreDir: t.TempDir(), so they have the LIFO problem too — the store directory removed underneath a live server. helpers_test.go doesn't set a store dir, so it's the milder version; I fixed it anyway for consistency.

One judgment call worth your eye: I put the full reasoning in startServer only, and a one-line pointer in the other three, rather than repeating an eight-line comment four times. If you'd prefer each site to stand alone, say so.

And I offered them the alternative I declined: a seam on the readiness wait (package-level var holding the wait func) would make it testable, but that's more machinery than the bug — and this repo's own package doc argues that a test which can't fail is worse than none. Their call.

Note the trail body's markdown table rendered as a flat paragraph — the trail document schema didn't parse it as a table. Cosmetic, but if that bothers you on these I'll use a bullet list instead next time.

Both fixed in 9fbd964, pushed to #102.

Finding 1 — my mistake, plainly. I registered defer stopProbe() then defer func(){ <-probeDone }(). LIFO means the join runs first and the cancel second, and the probe only exits on its context — so Run blocked in its own defer whenever it returned with the parent context still alive. One defer that cancels then joins.

The test uses the rejection this migration actually risks — a durable pre-existing with an immutable field set differently — and the broker obliges verbatim: ack policy can not be updated (err_code 10012). Verified it catches the bug: 20s hang with the two defers restored, 0.04s with the fix.

Finding 2 — also right, and the fix had a trap in it. The probe is the programmatic signal Run doesn't expose, so it now drives natslog.Health: SubscribeErr on failure, Recovered on success. That restores both losses — the 5-minute escalation and the restart-proof recoverable_failures counter — and it matters exactly where you said: US mode deliberately excludes this consumer from readiness, so telemetry is the only signal there.

But wiring it naively would have made things worse. nuts.IsTransientSubscribeErr keys on the legacy nats.* values, and jetstream.ErrStreamNotFound is a distinct *jsError — errors.Is finds neither across the two. So every rollout-ordering failure would have been classified a fault and logged at ERROR: precisely the alert-eroding noise natslog exists to prevent, introduced by the thing meant to prevent it. natslog now understands the modern errors, with ErrConsumerCreate/ErrBadRequest deliberately staying at ERROR so the migration's likeliest failure still pages.

Two things I got wrong along the way and fixed: I invented jetstream.ErrConsumerNotActive and ErrNoHeartbeat (neither exists — the build caught it), and my new test table had a wantMsg column nothing read, which hid a wrong expectation. That's the same defect as no test, so the message is now asserted.

The natslog extension belongs in go-nuts beside the predicate it extends. It's now the fourth item on that follow-up list, alongside Runner.Drain, FloorMonitor-attach, and Run-readiness — all four are the same shape: the library doesn't yet expose what a migrated consumer needs to stay observable. Worth raising as one go-nuts issue rather than four, if you want me to.

#102 is six commits. Gate clean, including -race -count=2.

Fixed in dd2cd57. Both findings were mine, introduced by the previous round's fix — worth saying plainly.

Finding 1. I demoted ErrConsumerNotFound inside natslog, and natslog already had the contract I broke: nats.ErrConsumerDeleted is kept at ERROR precisely because recreation loses the cursor. For this consumer that's concrete — DeliverPolicy: all on a stream retaining up to 96h, so a recreate re-fans the retained backlog cross-region. My change would have let that happen under generic warnings.

The real error: I put a decision in a predicate over error values when the information needed isn't in the value. natslog goes back to keeping a missing consumer loud; the probe decides, because only it knows which side of the first bind it's on. Before any bind → debug (Run is concurrently creating it and logs its own attempts). After a bind → SubscribeErr, now ERROR. Everything else → SubscribeErr from the first attempt, so a genuine boot outage still escalates at five minutes.

And I stated the limit rather than implying the probe is a detector: at 5s it can miss a delete that Run recreates before the next tick. The reliable signal is the runner's own recreate, which jsconsumer only logs.

Finding 2. Bounded to 20 attempts at 250ms (~5s), then steady regardless of success.

On the burst test — my first version re-implemented the cadence loop and counted its own probes over 8 wall-clock seconds. It passed, and would have kept passing with the bound deleted from production. Same defect as no test, and slow too. The decision is now nextProbeInterval, which probeDurable actually calls; verified it fails with the bound removed (interval = 250ms, want the steady 5s). 8.03s → 0.00s.

That's the third time this round that writing the assertion first exposed something: the wantMsg column nothing read, the duplicated cadence loop, and the fake that counted only successful publishes. Worth noting as a pattern rather than three coincidences.

#102 is seven commits, gate clean including -race -count=2. The go-nuts follow-up list is now five items, all the same shape — a migrated consumer can't stay observable without them: Runner.Drain, FloorMonitor-attach, Run readiness, modern-error classification, and a runner-sourced recreate signal. Still happy to raise that as one issue when you want.

Fixed in 316a5d7. That was the sharpest of the three, and the diagnosis was exactly right: I bounded the volume and left the classification unbounded, so the finding's scenario — creation failing while CONSUMER.INFO answers not-found — stayed silent forever. The gap the probe exists to close, still open after two rounds of me claiming to close it.

The startup budget now governs the grace too. The loop computes inStartupGrace (never bound and attempts remaining) and passes it in, so durableReachable no longer infers the situation from the error value — which it can't, since the same ErrConsumerNotFound means "racing the create" or "will never appear" depending only on how long it's been true.

Your point about the test was the important half. It set everBound by hand, so it structurally could not reach that path. The new one drives the real probeDurable and never touches it. One seam — probeStartupAttempts, zero uses the shipped constant — because the path requires the budget to expire, and 20 × 250ms is five seconds per assertion.

Two setup mistakes, both caught by the test failing rather than by me reasoning:

• I first pointed it at a broker with no stream. That yields ErrStreamNotFound, which already reaches Health as the canonical rollout line — a different path, and the assertion never matched. The finding needs the stream present and the durable absent.
• Verified it catches the bug: unbound the grace again and it fails with "stayed quiet past the startup budget; nothing entered natslog.Health".

One number I made explicit in the code because it's operator-visible and I'd only had it implicitly: with the shipped constants the first loud report lands ~10s after start (~5s grace, then the next steady probe). The alternatives were a fast cadence outliving its race, or a second knob to hurry the first complaint. Ten seconds to notice a durable that will never appear seemed well inside what matters — say if you disagree.

#102 is eight commits. Gate clean including -race -count=2.

Stepping back: this consumer now carries ~120 lines of readiness and telemetry machinery that exists purely because jsconsumer.Run exposes neither. Loops 2–11 each have a /readyz gate and the same telemetry need, so either that machinery gets extracted here and repeated eleven times, or go-nuts grows a Run that reports its own state and the app-side code mostly disappears. Given five follow-ups now point the same way, I'd rather fix go-nuts before loop 2 than build the second copy. Worth deciding before I start it.

Yes — CI was red, and it was my test, not production code. Fixed in 3765557; CI is re-running now and I'll report when it lands.

The race: captureLogs handed a bare bytes.Buffer to the probe goroutine (via slog) while the test polled buf.String(). bytes.Buffer isn't safe for that. My local -race -count=2 passed because the interleaving needs a write and a poll to coincide; CI's scheduling produced it, mine didn't. Now mutex-guarded, and I verified both directions — removing the mutex reproduces WARNING: DATA RACE locally on the focused test, restoring it passes at -count=3, and the full -race ./... suite (what CI actually runs) is clean.

Fixing it exposed a worse problem in the same tests. captureLogs replaces the process-wide default logger, so a parallel test's output lands in the same buffer — and my assertions searched the raw text for "level":"ERROR" anywhere in it. Another test's unrelated error would have satisfied that, and the quiet-path assertion could have failed on output that was never this consumer's. Both were passing for reasons unconnected to the behaviour under test.

Assertions now parse the captured JSON, find the record whose msg is the probe's own line, and read the level off that record — plus was_bound:false on the same record, so the two loud cases stay distinguishable instead of collapsing into "something logged an error".

Worth naming the pattern, since it's now four for four: every time I've written an assertion this round, checking whether it could fail found something — the unread wantMsg column, the duplicated cadence loop, the fake counting only successful publishes, and now a buffer race plus text-matching that could match anything. The local gate passing is not the same as the test being sound, and I've been treating them as equivalent.

One process note: I should have run the full -race ./... before pushing rather than -race -count=2 on selected packages. CI runs the whole suite, and the difference is exactly where this hid. I'll use the full suite from here.

To answer your question directly: no, I hadn't. Two more findings were open when you asked — one I was mid-fix on, one I hadn't seen at all. Both now fixed in 2f0b483.

The cadence one (which I'd just finished): a success settled the interval but didn't spend the budget, so a durable binding on its first probe kept ~19 unspent attempts, and a real loss within the next ~95s went back to 250ms — hammering a server that had just lost the durable. The comment three lines above said that must never happen; the code didn't implement it. Every single-call assertion I'd written passed, because the broken property is a sequence: end, and never resume. It's now a small state machine whose observe(bound) zeroes the budget, with sequence tests (never-binds / binds-then-loses / binds-mid-startup-then-loses).

The one I hadn't seen: context.Canceled isn't in the transient set, so every rollout logged a false ERROR — {"level":"ERROR",...,"error":"context canceled"}. Self-inflicted shutdown noise, which is exactly what natslog was built to remove, reintroduced by the probe I added to report real outages. Guarded narrowly on both parent-done and Canceled; the test asserts silence on a cancelled parent and a complaint on a live one, so the guard can't later be widened into swallowing real failures.

Verified both by breaking them: the cadence test fails with "a success must end the startup phase, not merely pause it", the guard test reproduces the exact ERROR record.

How I should have answered you. I've said "fixed" several times this round and been wrong twice. The reliable check isn't my recollection of the diff — it's review_comments_list plus CI, which is what I ran when you asked and what surfaced the open finding. I'll check that before claiming done rather than after you ask.

CI is running on 2f0b483; I'll report when it lands. The prior head 3765557 passed, so the race fix is confirmed green.

One thing worth naming: this loop has now taken five review rounds, and every finding has been in the readiness/telemetry scaffolding I added — none in the actual migration (ladder, disposition, config, spans). That machinery exists solely because jsconsumer.Run exposes neither readiness nor failure state, and loops 2–11 each need the same. I'd fix go-nuts before loop 2 rather than write the second copy of it.

Recommendation: fix go-nuts before starting loop 2, not after.

The problem. Loop 1 ended up carrying ~130 lines of production code plus 5 of its 10 broker tests on readiness and failure telemetry — probeDurable, durableReachable, probeCadence, two atomics, four constants, and the natslog modern-error extension. None of it is about the migration. All of it exists because jsconsumer.Run exposes neither am I consuming nor have I been failing, so the app reconstructs both by polling CONSUMER.INFO from the outside.

That's also where every one of the five review rounds landed — the ladder, disposition, config and spans drew no findings at all. Five rounds of defects in scaffolding that shouldn't need to exist.

Why it repeats. webhook-forwarder (loop 2) and pkg/fanoutengine (loops 4–10, eight durables) both have exactly the same subscribed atomic.Bool + Ready() + natslog.Health shape that loop 1 had. Migrating them means either copying this machinery twice more — including its bug surface — or extracting a shared in-repo package that goes stale the moment go-nuts grows the real thing.

What go-nuts needs. Four items, all the same shape, and one PR:

1. readiness from Run — did the current attempt bind
2. a failure-state hook — so natslog.Health can be driven by the runner instead of inferred
3. modern-client error classification (the jetstream.* values nuts.IsTransientSubscribeErr doesn't know)
4. Runner.Drain — separately needed for cmd/fanout's pool

With 1–3, most of loop 1's scaffolding deletes and loops 2–10 never grow it. Item 4 is what unblocks cmd/fanout adopting the scaffold at all, which would also retire the exit-criterion exception you accepted.

Cost. A go-nuts PR (small — it's exposing state the runner already has), a version bump, then a follow-up on loop 1 removing the app-side copy. Against that: writing the same machinery twice more, and reviewing it twice more.

The alternative if you'd rather keep momentum: proceed to loop 2 and extract loop 1's machinery into internal/consumerready as its second caller. Cheaper now, but it's a shim with a known expiry, and it doesn't help cmd/fanout.

My call: go-nuts first. The evidence is that this scaffolding is where the defects are, and I'd be writing it two more times before the library catches up.

Either way — loop 1 (#102) should merge on its own merits first; it's green and complete. CI on 2f0b483 is still running.

The middle result is suspect — my break script's indentation didn't match, so it likely changed nothing. Let me verify properly.