Add the embedded real-broker JetStream semantics suite (COR-1257)
Commit

Every P1 found in the A1 review cycle was a model error: code faithfully implementing a wrong belief about JetStream. The unit suites run against natsmsgtest.FakeMsg, which encodes the same model as the code, so it stays green exactly when the model is wrong — the only step in the pipeline that consulted reality was human review, which is why review kept finding P1s.
internal/brokersemantics asserts those beliefs against an embedded
in-process nats-server at the version pinned in go.mod. 21 tests, ~4s, and
deliberately NO build tag: it runs in the default go test ./..., so it
gates every merge. A tag CI forgets to pass is a gate that silently does
not run — the same failure shape as a fake agreeing with the code.
Three beliefs the suite falsified, each of which had a fix built on it:
-
Term SETTLES. ENT-1492 and COR-944 record that "a Term does NOT delete it (verified live on NATS 2.14.2)", and the fleet stream manifests still carry that note. It does not reproduce: the floor advances immediately and NumAckPending drops to zero on limits, workqueue and interest alike, and on the latter two the message also leaves the stream. Running the same test against an embedded 2.14.2 gives identical results, so it is not a version difference either. backoff's TermOnExhaustion promise (COR-762) holds as written; ENT-1492's premise for replacing Term with dead-letter-then-Ack does not — capture-first is still right, but for the record it keeps, not the settlement.
-
Ack, Nak and Term can SUCCEED WITHOUT SETTLING. They are fire-and-forget publishes to $JS.ACK.>, so with an identity lacking that publish grant — the gap COR-1224 found on five live consumers — every disposition returns nil while the server rejects it, the floor stays pinned, and nothing is logged. The rejection reaches only the connection's async error handler. This is a mechanism that produces ENT-1492's symptom without requiring Term-does-not-settle to be true. DoubleAck does report it, as context.DeadlineExceeded rather than nats.ErrTimeout, so a classifier built from nats sentinels alone misses it.
-
A NakWithDelay is STRETCHED BY THE LADDER. The server offsets the delivery timestamp by the requested delay but still measures it against the ladder's current rung, so what it serves is requested + (BackOff[rung] - BackOff[0]), rung clamped to the last entry. Measured to the millisecond. A client-side delay therefore cannot express its own envelope on a growing ladder, and this is the arithmetic behind ENT-1535's unexplained retry spacing.
Also pinned, in the terms a config review and a runbook need: the ladder rule (the wait before delivery N is BackOff[min(N-2, len-1)], so MaxDeliver-1 waits, the last rung tail-repeating, and a rung that is dead configuration when MaxDeliver == len(BackOff)); NumDelivered climbing to MaxDeliver with no handler involvement at all; an exhausted message pinning the floor with NumAckPending ZERO, so "Outstanding Acks: 0" does not mean nothing is outstanding; the floor advancing on stream-side removal with no ack, after which floor+1 names an undelivered or nonexistent sequence (the ENT-1535 break-glass); on a filtered consumer floor+1 naming an innocent unmatched sequence until an unrelated ack below the blocker flips it, which is what makes it unusable as a blocker identity; AckWait silently rewritten to BackOff[0] on the ordinary client path but rejected under pedantic mode, which NACK's controller uses for fleet Consumer CRs; len(BackOff) > MaxDeliver rejected while == is accepted, the server's own error text overstating its rule; a stream's ConsumerLimits filling a silent consumer's zero MaxAckPending and InactiveThreshold; and the library's end-to-end claims — TermOnExhaustion firing on the broker's real final delivery, KeepInProgress holding a delivery past AckWait and releasing it after the cap, DeadLetter's capture-then-Ack, and a durable resuming rather than replaying.
natsmsgtest is demoted in its own doc to library logic only: what did the code DO, never what the broker does in response. README documents the split and how to read a failure after a server bump.
Two things the suite guards about itself. A fixture whose parameters are degenerate measures the semantic away and then agrees with the code for the same reason a fake does — a flat 3s/3s/3s ladder is green against a live server and pins no stretch at all — so doc.go states the rule for adding cases. And TestMain names the server version on stderr when a run fails, because the merge gate is non-verbose and a t.Logf in a passing test would be discarded.
The ireturn allow-list gains jetstream.Stream and jetstream.Consumer, alongside the jetstream.JetStream entry already there for the same reason.
Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com Claude-Session: https://claude.ai/code/session_01WNGoni5q31iGD4KoWvFJ93 Entire-Checkpoint: 851f01f4a88c