natsmsg: stop deduping captures; make origin provenance write-once

main

Commit

nodo3w ago

Direction change from Andrea, superseding the identity-key iteration: collapse the requirement rather than refine the key.

NO CAPTURE-TIME DEDUPE. A captured copy now goes to the DLQ with no Nats-Msg-Id at all. The dedupe existed for one narrow case — the DLQ publish lands, the original's Ack does not, the message redelivers and is captured again — and every identity that can be synthesized for it has a collision class, each found in review rather than by reasoning ahead: the publisher's own ID collapses distinct originals because JetStream dedupes on it stream-wide; origin stream plus sequence collapses across a recreated stream; the next scheme would meet the next broker behaviour.

The requirement gives because the two failure directions are not comparable. A second stored copy is VISIBLE — a record someone can read and reconcile, and at-least-once already puts duplicates everywhere else in JetStream. A suppressed capture is INVISIBLE: the broker answers with a successful PubAck marked Duplicate, the caller reads success and Acks the original, and the message is gone with no record, defeating the one guarantee this path exists to provide. Same choose-the-safe-direction principle as the breaker's clock — late, never early; here, duplicate, never suppressed. The cost is an extra record in the failed-Ack window, bounded by the deliveries left on the ladder. Replay-side dedupe has the origin provenance and far more context than a publish-time key affords.

WRITE-ONCE ORIGIN PROVENANCE. Origin headers are filled in only when absent, so capturing a message twice never overwrites what the first capture recorded — the failed-Ack re-capture, and a replay tool dead-lettering a DLQ record it gave up on. Origin now also records the store timestamp and the JetStream domain, which the retired key was carrying and which replay-side dedupe needs to tell two incarnations of a recreated stream apart. Per-hop fields are the opposite: reason and delivery count describe the capture that just happened, with Nats-Dlq-Hops counting captures.

Retention is a closed list of the origin keys this package defines, NOT the Nats-Dlq- prefix: an invented Nats-Dlq- field never rides through. One limit is stated rather than implied — no header-level rule can tell an origin header written by an earlier hop from one a producer set, so origin headers are what the capture chain REPORTED. What is guaranteed is the shape: only these keys appear, per-hop fields are always this library's, and the prefix alone buys nothing.

Broker-backed, all three the review asked for:

  • source recreation — the stream is deleted and recreated so the second message is stored at sequence 1 like the first, and both records exist;
  • the general invariant — four captures, two distinct originals plus two re-captures of one, leave four records, on a DLQ whose duplicate window is wide enough that any identity would have collapsed them;
  • re-capture preserves first-hop provenance. Worth naming: re-capturing an ORIGINAL cannot discriminate write-once from overwrite, since both derive the same values from the same stored message. So that test also captures a DLQ RECORD, where this hop's stream, sequence and subject all differ — which does discriminate, and fails under overwrite.

The fake publisher keeps modelling JetStream's dedupe even though copies no longer carry an ID: it is now the GUARD that makes the no-dedupe property testable. Reintroduce an identity and the exactly-N tests fail, which is the check the old design lacked.

Refs ENT-1601, ENT-1535, ENT-1492, COR-1257.

Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com Claude-Session: https://claude.ai/code/session_018qXhU99D2Sxaa32yZWwYRP Entire-Checkpoint: 2e68e099bf66

Checkpoints

Reconcile jsconsumer Retry Schedule After Broker Semantics

Claude CodeOpus 5.[1m]
View session
Checkpoint 1