NATS Robustness Consume Loop Restart

I’ll pin COR-1252’s current Linear state, then reconcile it with the six repo checkouts and any live PR/CI evidence so the status reflects what is actually landed versus still pending.

The umbrella is still marked In Progress, and its last update (Aug 14) says the initial stall-remediation track shipped, while the standardization/guardrail work remains open. I’m now checking each named track and the actual branches/PRs, because the six local checkouts are all clean on main and do not themselves show whether the work merged elsewhere.

Live delivery evidence confirms the first layer is genuinely merged: go-nuts’ retry/real-broker work, the entire-search adopter, the Fleet runbook, and Infra monitor changes are all on main. Track D has only reached its bootstrap plus the first consumer migration draft; I’m checking whether that draft has review blockers or any newer, unlinked work before giving you the bottom line.

As of Aug 17: the foundational fix shipped, but the standardization program is still early. I would not call COR-1252 done.

What shipped:

• go-nuts retry/DLQ/floor telemetry: #14.
• Real-broker semantic suite: #15, #16.
• entire-search adoption: #178. Its staging deployment and production promotion succeeded, and the generated Fleet promotion #4436 merged. I did not re-check actual pod reconciliation or post-rollout production behavior.
• Runbook and monitor changes: Fleet #4398, Infra #1167.
• Track D bootstrap/harness: mirror-pipeline #245.

The immediate work item is mirror-pipeline #246: open draft, green CI, clean merge state, no formal reviews. It is the first actual consumer migration; roughly ten durable migrations remain afterward. Nothing newer is linked, and activity stopped on Aug 14.

Important scope drift:

• Consumer CR ownership was parked after the canary analysis found bind-only ordering and InactiveThreshold blockers. COR-1252 and COR-1254 still describe CR-managed durables as the plan, so their descriptions are stale.
• The enforcing floor breaker did not ship operationally: it is observe-only, its soak never started, and enforce-or-delete remains undecided.
• The live poison-message conformance canary was canceled, leaving the end-to-end DLQ path unproven in a live cell.
• Track E’s “generate grants and monitors from Consumer CRs” now needs redesign or parking.

Known residual risks include COR-1270 — a high-priority live 41-hour orphaned ladder — and ENT-1496, the lifecycle consumer’s remaining bare-Term behavior. The non-lossy quarantine tool was canceled, so a capture-defeating stall still falls back to lossy stream rmm.

Bottom line: one strong library/adopter slice is in production desired state; the “stop fixing per-app” outcome is not yet achieved. The critical path is review/merge #246, burn down the remaining consumers, then rewrite Track E around the non-CR paved road. M3 remains recorded for Oct 14 and M4 for Oct 30, but the umbrella has no due date or current health signal.

I’m treating this as a live correctness review: I’ll re-pin trail 102’s current head, inspect the full diff and adjacent jsconsumer/JetStream lifecycle contracts, then run focused validation where the checkout permits it. I’m using the prior live-PR review playbook because this change crosses real-broker retry, shutdown, and telemetry semantics.

The live head is still 2f0b483fd622a6d3333eba164ef29e491c7d2947, based on 1b24bd6; it is mergeable, draft, and CI-green. That head includes fixes for the earlier race, one-way probe cadence, and shutdown-noise findings, so I’m reviewing the resulting state afresh rather than carrying those findings forward.

The central migration invariant—no disposition on transient failure so server BackOff owns redelivery—is implemented correctly and covered against a real broker. I’m now pressure-testing the parts that can still invalidate “behavior preserved”: handler duration versus AckWait, multi-replica overlap, durable recreation, and readiness/telemetry transitions.

Request changes. I found one P2 correctness issue at head 2f0b483f; the head remained stable through review.

Finding — retry exhaustion now happens earlier

The migration changes when the retry timer begins:

• Previously: fan work completed, then NakWithDelay(30s) started the delay.
• Now: JetStream’s 30-second BackOff begins at message delivery, overlapping fan processing.

Fan-out is sequential across three remotes, each with a five-second timeout (fan loop, publish timeout). Failed attempts are intentionally left unsettled for server redelivery (failure path).

A real-broker diagnostic measured the difference with a 400ms handler and one-second delay:

• Old client-owned delay: 1.402s
• New server-owned delay: 1.002s

In production, one timed-out remote shortens the eight-delivery recovery window by roughly 35 seconds; three timeouts can shorten it by roughly 1m45s. Because there is no terminal DLQ, earlier exhaustion can turn a recoverable regional outage into missed fan-out requiring manual replay.

The timing tests currently compare only configured delays and use an immediately failing publisher, so they cannot detect this change (timing assertion, broker fixture).

Either preserve the old wall-clock recovery window, or explicitly accept and document the shorter envelope and add a slow-handler real-broker test.

Validation otherwise looks good: focused race tests passed, touched package tests passed, diff --check is clean, CI is green, and Fleet’s current lifecycle/JetStream permissions cover the new consumer. The earlier race, readiness-cadence, and shutdown-log findings are fixed.

I’ll own the decision against the change’s actual objective, not just patch the symptom. I’m re-pinning the trail and checking the timing contract plus operational consequences; then I’ll recommend the concrete path I’d ship and whether it belongs in this PR.

The key constraint is now clear: the migration must keep JetStream as the single retry owner. Reintroducing NakWithDelay would fix the timestamp arithmetic by recreating the exact mixed-ownership failure this track exists to remove. I’m checking whether the server model can preserve the old wall-clock window cleanly, or whether the honest fix is to accept and encode the timing change.

I recommend accepting the clock-origin change and making it explicit. Keep the server-owned 30s × 7, eight-delivery ladder.

Exact wall-clock preservation is incompatible with this migration:

• Restoring NakWithDelay recreates mixed ownership.
• Sending InProgress after failure is client scheduling disguised as a heartbeat.
• Increasing the rung to 45 seconds only approximates today’s three-region topology and unnecessarily slows fast failures.

I would add one corrective commit to this trail:

• Replace “timing envelope unchanged” with “scheduled ladder unchanged; wall-clock recovery window is shorter by handler time.”
• Fix timingcontract terminology so its t+ values are clearly scheduled offsets, not guaranteed delivery timestamps.
• Add a slow-failure real-broker test proving the 30-second server clock starts at delivery.
• Record the operational delta: current production handler bound is approximately 15 seconds, so the eighth attempt moves from 3m30s–5m15s previously to approximately 3m30s.
• State that no existing monitor or runbook encodes the old wall-clock value, so no companion configuration change is required.

I would not expand the ladder or add a DLQ in this PR. Those are policy changes, whereas this trail should establish one honest, server-owned timing model. After that corrective commit passes CI, I’d consider the trail code-ready.

I’m applying the agreed correction on the live trail: one commit that keeps the runtime ladder unchanged, fixes the shared scheduled-time terminology, and adds slow-handler broker evidence. I’ll run the focused race suite and full repository gate before pushing.

The workspace’s custom Git remote has an expired Entire login, but GitHub access is healthy. I’m using the authenticated GitHub transport for this branch; the repository contents and target PR are unchanged.

The correction is now in code. Runtime policy remains unchanged; the diff only makes the accepted clock-origin delta explicit and adds a broker test where a 500ms handler cleanly separates server timing from the legacy client timing. I’m validating that test for stability before running the broader gate.

The new broker test passed five consecutive runs and the focused race suite passed three consecutive runs. The rendered table now says exactly what it measures—scheduled offsets—and no longer claims wall-clock equivalence. I’m moving through build, vet, lint, module consistency, and the full race suite now.

Lint and module consistency are clean. The parallel build/vet invocation hit a transient vendor-resolution failure while the same tree was being analyzed concurrently; the vendor files are present and the focused packages already compiled. I’m rerunning build and vet serially to distinguish tool contention from a real repository problem.

All gates are green: build, vet, lint, module consistency, full tests, and full -race. The accepted delta is now executable evidence rather than prose alone. I’m doing the final diff review, then I’ll commit through the workspace hook and push the trail branch.

The commit hook did not add the required Entire-Checkpoint trailer, so I’m not pushing it in this state. Per the workspace instructions, I’m running the workspace doctor and will repair the hook, then amend without manually typing any trailer.

The corrective commit is pushed as a574849c with checkpoint 53ef0b702b0c; PR 246 now points at that exact head. GitHub CI has started, so I’m holding ownership through the remote result rather than stopping at the local green run.

Remote CI has completed build, vet, the full race suite, and formatting successfully; only golangci-lint is still running. The new broker timing test has therefore passed on GitHub’s runner as well.

Implemented and pushed to PR #246.

• Commit: a574849c
• Entire checkpoint: 53ef0b702b0c
• Kept the server-owned 30s × 7 ladder unchanged.
• Corrected timing output to distinguish scheduled offsets from wall-clock guarantees.
• Added real-broker slow-handler coverage proving BackOff starts at delivery.
• Build, vet, lint, full tests, full race suite, and GitHub CI all pass.
• PR is clean/mergeable and remains draft.
• Working tree is clean on the PR branch.

In simple terms, this PR standardizes how lifecycle events are copied between regions.

The problem

meta-fanout reads lifecycle events from the local NATS stream and copies them to every other region.

Its consumer was hand-written:

• The application managed retries itself.
• NATS also had retry-related configuration.
• Those two sources of truth could disagree, producing surprising retry timing—the same class of problem that previously caused a 48-hour backlog.
• Consumer configuration was partly implicit and harder to test or inspect.
• Readiness and failure reporting were custom and inconsistent.

The solution

The PR moves this loop onto the shared go-nuts/jsconsumer framework:

• NATS becomes the single owner of retry timing.
• A failed regional publish is left unacknowledged; NATS redelivers it.
• A successful fan-out is acknowledged.
• The policy remains eight attempts with 30-second scheduled intervals.
• The existing durable name is preserved, so its cursor and pending work are not recreated.
• The consumer configuration is explicit and tested against a real NATS server.
• Readiness now verifies that the durable is actually reachable.
• Existing failure telemetry and shutdown behavior are preserved.

One timing detail changes deliberately: previously, the 30-second delay started after processing finished. Now it starts when NATS delivers the message, so retries can happen slightly sooner. The PR now documents and tests that honestly.

In short: we replace a bespoke, drift-prone retry loop with one standardized consumer where NATS clearly owns retries and real-broker tests prove the behavior.

I’m checking this as a rollout-contract finding, not assuming the embedded tests prove credentials. I’ll verify the exact JetStream API subject requested by checkAgainstStream and the live Fleet grants used by meta-fanout across the seven cells; if the wildcard already covers it, I’ll close the finding with evidence rather than change code.

The concern is technically valid, but the rollout is already covered: checkAgainstStream publishes $JS.API.STREAM.INFO.resource_lifecycle_v1, while Fleet grants meta-fanout $JS.API.> plus _INBOX.> in the shared NATS base. That base is used directly by all three staging cells and renders unchanged through all four production overlays. I’m checking whether the trail finding is attached as a replyable review thread so I can close it with that evidence.

This finding is resolved; no code or Fleet change is needed.

The concern is correct: checkAgainstStream requests:

$JS.API.STREAM.INFO.resource_lifecycle_v1

But Fleet’s shared meta-fanout identity already grants:

• Publish: $JS.API.>
• Subscribe: _INBOX.>

See Fleet’s NATS configuration.

That base configuration is used directly by all three staging cells, and I rendered all four production overlays to confirm they retain the same permissions. Therefore all seven cells can perform STREAM.INFO and receive its response.

It also would not fail silently: jsconsumer.Run logs every failed start attempt, /readyz stays false, and the lifecycle health supervision escalates after startup grace.

Verdict: valid dependency to verify, already satisfied. PR #246 remains unchanged and green at a574849c.

Correct—nothing to change for this PR.

The new API call is already covered by Fleet’s $JS.API.> grant in all seven cells, with _INBOX.> for responses. The consumer also fails visibly if that permission ever disappears.

Optional future hardening: add a Fleet CI assertion for required JetStream subjects. I would not hold or expand this PR for it.

Noted—PR #246 is merged. The lifecycle consumer migration is complete with server-owned retries, explicit timing semantics, and green real-broker coverage.

I’ll follow this through build, Fleet promotion, all seven meta-fanout deployments, and the first post-rollout health window. I’ll treat “done” as: expected image live everywhere, pods Ready without restart churn, lifecycle durable healthy, and no new consumer/fan failure signal.

The new meta-fanout image has built and pushed successfully. I’m now waiting on the deploy/Fleet promotion stage; once its digest is published, I’ll trace that exact image through all seven cells.

The deploy commit is now in Fleet with tag main-1321264, pinned to the merged source SHA. It intentionally updates the three staging overlays first; I’m watching the automated promotion that resolves the digest and advances production.

Promotion is healthy but not merged yet: Fleet PR #4517 has auto-merge enabled and is waiting only on its lint job. All three staging cells remain healthy on the previous image while Flux waits for the new desired state—no partial rollout yet.

Fleet promotion #4517 passed lint and auto-merged at 08:53:16Z. Production is pinned atomically in all four regions to digest sha256:1f73f3e0…47cc4fcf; I’m switching to cluster-by-cluster verification now.

Rollout is active: six of seven cells have accepted the new image. Four are already at full desired readiness; EU staging, US production, and AP Southeast are still draining their old ReplicaSets. Staging US West is the only cell still awaiting its Flux turn. No availability drop so far.

I found an actionable rollout problem in AP South: both new pods are running but 0/1 Ready, and the old pods have already drained, so that cell’s deployment currently has zero available replicas. I’m pulling the startup/readiness logs now to identify whether this is the ACL/stream-info concern or a different migration failure.

Continuing. I’ve pinned the cause: the legacy PullSubscribe consumer was created by the old client, and nats.go explicitly deletes such consumers when that client drains. In AP South, the second old pod drained after the new pods had rebound, deleting the durable a second time; the new consume loops then stayed alive but deaf. I’m checking whether it recovered and whether any other cell hit the same terminal state.

Rollout is healthy in 6/7 cells, but AP South is stuck at 0/2 Ready.

This is not the ACL concern. The new consumer initially started and processed events. During the mixed-version rollout, the draining legacy pods deleted the shared durable; the new consume loops stayed alive but deaf and now repeatedly report consumer not found.

Recommended recovery: sequentially recycle the two AP South pods—wait for the replacement to become Ready before recycling the other—then verify the durable, backlog processing, and health window.

May I perform that recovery?

I’m checking the cell live, then I’ll separate the rollout mechanics from the actual impact.

It’s not the whole AP South cell. The failure is isolated to its meta-fanout lifecycle consumer.

What happened:

1. Both new pods started successfully and connected to local and remote NATS.
2. During the rolling deployment, an old pod shut down. The legacy PullSubscribe client deletes the durable consumer when its connection drains.
3. The new pods noticed that first deletion and recreated the durable.
4. The second old pod then shut down and deleted the newly recreated durable again.
5. This time the new pull loops remained open but received only no responders/heartbeat errors. Because they never formally closed, the supervisor didn’t recreate the consumer.
6. The readiness probe correctly detected that meta-fanout-lifecycle no longer exists and marked both pods NotReady.

Current state:

• New image is deployed.
• Both processes are Running, with zero restarts.
• Both are 0/1 Ready; deployment is 0/2 Available.
• The lifecycle durable is still missing.
• AP South lifecycle events are therefore not being fanned to other regions. They remain in the local stream for its retention window, so this is currently a growing backlog rather than immediate loss.

AP South alone lost the timing race: its second legacy pod drained after the new durable had already been recreated. Other cells either completed both old-pod drains before their final recreation or otherwise avoided that ordering.

Two fixes: recover AP South now, then close the robustness gap.

Immediate recovery:

1. Recycle one of the two stuck AP South pods.

2. Wait for its replacement to create meta-fanout-lifecycle, become Ready, and process events.

3. Recycle the other original pod as well. This matters because it may appear Ready once its peer recreates the shared durable, while its own consume loop remains deaf.

4. Verify:

   • both original pod UIDs are gone;
   • deployment is 2/2 Available;
   • the durable has the expected BackOff/MaxDeliver configuration;
   • backlog is processing;
   • no consumer not found, heartbeat, or no responders errors during the health window.

No legacy pods remain, so the durable should no longer be deleted during these restarts. I would not roll back—the old implementation is what deletes the durable during shutdown.

Permanent fix:

• Update go-nuts/jsconsumer.Run so no responders or heartbeat failures trigger a ConsumerInfo check. If the durable is missing, stop the stale consume context and recreate it instead of waiting forever for Closed().
• Add a real-broker regression test with two runners where the durable is deleted twice—the second deletion occurring after the first recreation.
• Make readiness represent the local consume loop, not merely “some replica recreated the shared durable.”
• For future migrations from legacy PullSubscribe, use a two-release transition: first make the legacy client bind an explicitly managed durable, then migrate to jsconsumer. That prevents old pods from deleting shared state while draining.

The immediate recycle restores service; the library fix prevents this failure mode from recurring.

No—this specific failure is a one-time mixed-version migration race.

The old implementation owns the durable implicitly, so draining an old pod deletes it. The new implementation creates the durable explicitly and does not delete it when a pod shuts down. Once AP South’s two legacy pods are gone, normal new-version → new-version deployments should preserve the durable.

However, we exposed a separate robustness gap: if anything deletes the durable later—operator action, stream recreation, controller issue—the current consume loop may remain stuck instead of recreating it. That is not expected every deploy, but we should still fix it in go-nuts.

So:

• Recycle AP South now: resolves the one-time transition failure.
• Fix jsconsumer.Run: protects against future external durable deletion.
• Don’t roll back to the legacy version, or we reintroduce the delete-on-drain behavior.

I’ll recycle them one at a time. I’ll wait for the first replacement to create the durable and become genuinely healthy before removing the second original pod, then verify both replacements and the backlog.

The normal production context is read-only and rejected the deletion, so nothing changed. I’m switching to the configured AP South admin context, revalidating the same pod UID there, then I’ll perform the authorized recycle.

The first replacement (…cch9h) recreated the durable and became Ready. As expected, that also made the untouched stale pod appear Ready because both probes see the shared durable; I’m verifying the replacement is consuming before removing that remaining original pod.

Both original pods are now gone. Their replacements (…cch9h and …ljtdw) are Running, Ready, and at zero restarts; the first is actively replaying the retained lifecycle backlog. I’m validating the second consumer attachment and the stored durable configuration next.

Both replacements are consuming cleanly; the read-only natsctl verification was denied by the normal production role before creating anything. I’m rerunning that inspection through the AP South admin context so we can confirm the server-stored retry ladder and backlog rather than trusting readiness alone.

Recovery is complete at the broker level: the durable has the intended 30s × 7 BackOff, MaxDeliver=8, MaxAckPending=1000; its replay backlog is already fully drained (14,550/14,550 acked, 0 pending, 0 redelivered). Both replacement pods are actively consuming with no failure signals. I’m keeping the health watch open through the five-minute escalation window.

Fleet-wide check is clean: all 15 meta-fanout pods across seven cells are Ready, zero restarts, and no lifecycle-consumer failure signals in the recent window. AP South remains 2/2 Available; I’m holding until its five-minute post-recovery mark before closing the rollout.

The five-minute recovery window is clean. AP South has processed new events since the backlog drain; its delivery and ack floors are equal at 14,573, with 0 pending, 0 ack-pending, 0 redelivered, and two active pull waiters. I’m closing the rollout monitor as successful-with-recovery.

AP South is recovered and the rollout is complete.

• Recycled both stale pods sequentially; Kubernetes replaced them successfully.
• AP South: 2/2 Ready, zero restarts, clean five-minute health window.
• Durable config verified: 30s × 7 BackOff, MaxDeliver=8, MaxAckPending=1000.
• Delivery and ack floors match at 14,573; zero pending, ack-pending, or redelivered messages.
• All seven cells are on main-1321264; all 15 pods are Ready.

The permanent jsconsumer.Run self-recovery gap remains a follow-up. Monitoring completed in about 13m 34s.

I’m checking live flow now—not just pod readiness: exact image, restarts, recent consumer/fan errors, and each cell’s JetStream durable for pending or redelivered work.

Runtime is clean in every cell: all 15 pods Ready, zero restarts, correct image, and no consumer/fan failures in ten minutes. The app logs are quiet—no lifecycle events happened in that interval—so I’m checking the seven broker durables directly for waiters, backlog, ack-pending, and redeliveries.

Yes—live checks show everything is flowing normally now:

• All seven cells on the expected image.

• All 15 pods Ready, zero restarts.

• No consumer, fanout, heartbeat, or NATS errors in the last ten minutes.

• Every durable has active pull waiters matching its replica count.

• All seven report:

  • 0 pending
  • 0 ack-pending
  • 0 redelivered
  • delivery floor equal to ack floor
  • correct 30s × 7 retry ladder and MaxDeliver=8

AP South has continued advancing since recovery and remains fully caught up. The permanent self-recovery improvement is still worth implementing, but there is no current operational issue.

Next should be the robustness follow-up we just discovered before migrating another consumer. I’m checking the parent issue and local task sequence so I can give you the exact next slice rather than a generic cleanup list.

Next should be a short stop-the-line hardening slice before Loop 2.

1. Fix the production finding in go-nuts:

   • jsconsumer.Run must recreate a missing durable even when nats.go emits no responders/heartbeat failures without closing the consume context.
   • Readiness must represent the local consume loop, not merely a durable recreated by another replica.
   • Add a real-broker regression with two runners and two timed consumer deletions, matching AP South.

2. Release go-nuts, bump mirror-pipeline, and roll meta-fanout again normally.

3. Then proceed with the planned Loop 2: migrate webhook-forwarder. That should also resolve COR-1270:

   • replace its accidental 41-hour ladder with an explicit policy;
   • I recommend roughly 42 minutes—enough to absorb routine outages without pinning the ack floor for nearly two days;
   • add the missing ack-floor-age monitor in the same change.

After that, the approved sequence is the default-off engine switch, remaining low-risk consumers, and cmd/worker last.

I’d make the go-nuts recovery fix a blocking child of COR-1254 and start there.

We can leave the current deployment running; it is healthy, and the mixed old/new deletion race will not happen on ordinary future deploys.

But we should fix the library before migrating another consumer.

Why:

• jsconsumer.Run promises recovery when its durable disappears; production proved that promise is incomplete.
• A future operator deletion, stream recreation, or controller event could trigger the same stuck state.
• Another replica can recreate the shared durable and make a deaf pod appear Ready.
• Standardizing more consumers now would replicate that weakness across the fleet.

So my recommendation is:

• No emergency rollback or hotfix.
• Keep the current rollout.
• Treat the go-nuts recovery change as a blocker for Loop 2.
• Until fixed, the operational recovery is sequential pod recycling.

It’s safe to leave briefly; it’s not a good defect to knowingly multiply.

I’ll fix this in go-nuts at the supervision layer and add a real-broker regression matching AP South: two runners, a durable recreated after deletion, then deleted again while both consume contexts are live. I’ll first pin the current branch and existing lifecycle contracts, then drive the fix with the failing test.

The regression will model the actual mixed-version handoff, not just an administrative delete: an old PullSubscribe owns the durable, two new supervised runners bind to it, the durable is recreated once, and then the stale legacy subscription drains and deletes the replacement. That second deletion is the path that left AP South deaf.

The failing broker test now reproduces the important state exactly: after reconnect, the consume loop logs nats: no responders available for request, its Closed() channel never fires, and the durable never returns. I’m wiring supervision to the existing asynchronous consume-error path, with an authoritative check only when an error occurs—there will be no steady polling.

The fix is now green under a ten-run stress pass. It restarts only on ErrNoResponders/ErrConsumerDeleted; it deliberately does not restart on ordinary heartbeat or leadership-change notices. That avoids adding the extra CONSUMER.INFO ACL dependency we were worried about during the rollout. I’m moving through the full package/race suite now.

Everything is green now: lint, the full suite, and go test -race ./.... The regression also passes ten consecutive broker restarts. I’m doing the final diff/branch check, then I’ll commit and push this as the go-nuts hardening PR branch.

The commit hook did not add the required Entire-Checkpoint trailer, so I’m pausing the push and running the workspace doctor as instructed by this task’s AGENTS.md. I won’t hand-edit the trailer.