Recognise the rest of the transient pull-consumer failures

main

Commit

nodo3w ago

IsShutdownFetchErr matched only ErrConnectionClosed and ErrConnectionDraining, but a real drain rarely ends on one of those. A rolling pod restart mostly produces ErrNoResponders, and a NATS server upgrade produces ErrFetchDisconnected and ErrConsumerLeadershipChanged. Callers gated their error log on the predicate, so every one of those was logged as a fault.

Measured on one mirror-pipeline deploy: a single pod emitted 57 ErrNoResponders inside 83 milliseconds as its in-flight fetches failed together, and two services between them crossed a cluster's error-log alert threshold three times in forty minutes. The alerts were correct about the count and wrong about the meaning, which is the failure mode that teaches operators to skim the one signal that should mean something (COR-1226 / COR-1228).

Split the judgement in two. IsTransientFetchErr answers "is this recoverable", independent of shutdown — a server rolling underneath a healthy caller produces these while ctx is still live, so no ctx-based predicate can classify them. IsShutdownFetchErr becomes exactly "transient AND ctx done", which widens it correctly and leaves one list to maintain instead of two overlapping ones. A test pins that relationship so the predicates cannot drift apart again.

ErrTimeout stays out: an idle poll reaching MaxWait is the steady state of a quiet consumer, and callers already continue on it. context.DeadlineExceeded stays out too — a slow server and a wedged one raise the same error, so swallowing it would hide real stalls.

Callers should now log transient-but-running below error level and keep looping, reserving error for failures that stop recovering.

Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com Claude-Session: https://claude.ai/code/session_01HnPd4sZdPSrJXXJMEEfnMC Entire-Checkpoint: 3ac99c508950

Checkpoints

Fix Datadog Go slog Severity Mapping

Claude CodeOpus 5.[1m]
View session
Checkpoint 1