jsconsumer: one retry schedule, dead-letter capture, and floor telemetry

main

Commit

nodo3w ago

ENT-1535's stall had two causes, and only one of them was the poison message. search-indexer-refs advertised a 5m/10m/30m/1h/4h/12h ladder while the forensics measured a real 34h22m envelope: the handler's NakWithDelay and the consumer's BackOff were both scheduling redelivery, so the server stretched each NAK by the BackOff increments and neither ladder was the truth. A configured schedule that lies is how a single unindexable ref event pinned an ack floor for ~10h with nothing in the system saying why.

So this makes the schedule singular and checkable, and gives a consumer a non-lossy way to give up.

Schedule the retry timing as plain values, and the ONE implementation of the arithmetic that says whether it hangs together. Pure — no connection, no I/O, no clock — so the same function runs at Start, in fleet CI's admission lint over a rendered Consumer CR, and (later) at bind time against the durable's live config. Duplicated timing maths is how ENT-1535 happened; compile this in rather than restating it. Retry where retries END: dead-letter capture, then settle. The SERVER owns when they happen, through the durable's BackOff; Retry plain-Naks into it and has no ladder fields at all, so two competing schedulers are unrepresentable rather than merely rejected. FloorMonitor the ack-floor stall signal, as telemetry. Separate from disposition because ack-floor state is consumer-global while giving up on a message is message-local.

Never a drop and never a bare Term (ENT-1492): a failed capture Naks instead, CaptureReserve holds deliveries back to retry it, and when even those are spent the settlement reports OutcomeStranded rather than dressing it up as a retry — a Nak at the broker's cap is dropped, so nothing will touch that message again. The terminal Ack is a bounded DoubleAck, because a lost fire-and-forget Ack leaves the floor pinned with nothing reporting it.

The floor-age circuit breaker is here, wired to FloorMonitor, and is EXPERIMENTAL. BreakerObserve is the zero value and the only supported mode: it measures what enforcing would do, spending the same one-per-stall-window budget so the counts are a faithful dry run. Every serious defect found in review was in the enforce path, all from one root — acting on a client-side inference about which message holds a consumer-global floor. Deleting it is an acceptable outcome of the soak if bounded ladders prove sufficient.

Notable behaviours worth knowing about:

  • Stall age is measured while the consumer has delivered PAST its own floor. An idle consumer's motionless floor is not a stall, and charging the next arriving message with that idle time would quarantine it on delivery two.
  • Elapsed failure time is measured, not inferred from NumDelivered, which JetStream also increments on AckWait expiry — a panicking handler can burn four deliveries in milliseconds.
  • Schedule.TimeToDeadLetter repeats the last backOff rung once the array runs out, as the server does. A short array under a larger maxDeliver is not a short ladder, and under-counting would pass a config that breaches its bound in production.
  • Process dead-letters an undecodable payload instead of Terming it when a Retry is configured, closing the last drop-without-a-record surface.

Explicit non-goal: a handler that DIES on the poison message rather than returning an error. Settle is the only entry point to every disposition here, so such a message is not settled by any of them — a property of the callback contract, not of the breaker. Recovering the panic to Ack it would hide the bug and leave handler state unreconciled, against ShutdownGroup's existing fatal-panic posture. Detection is unaffected; remediation is not attempted.

Refs ENT-1601, ENT-1535, ENT-1492, COR-762.

Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com Claude-Session: https://claude.ai/code/session_01AJNdqGnT92NC2TRnpf8MNr Entire-Checkpoint: 52c7272fa3e6

Checkpoints

Adversarial Review of Retry and Dead-Letter Changes

CodexGPT-5.6-sol
View session
Checkpoint 1

Fix NATS Consumer Retry Ladder Implementation

Claude CodeFable 5
View session
Checkpoint 1