Fix Datadog Go slog Severity Mapping

All three done.

Committed — mirror-pipeline branch nodo/cor-1226-..., commit f5764ec, local only (not pushed). The post-commit hook attached Entire-Checkpoint: 69c61b2143d1 automatically. Build, tests (17 packages), and golangci-lint all clean.

COR-1226 — findings comment posted, covering the narrowed root cause, why no pipeline work is needed, the monitor audit, and the predicted surge-monitor noise.

COR-1228 — Transient NATS reconnects are logged at ERROR across mirror-pipeline services, assigned to you, Backlog, linked to COR-1226. Recommends re-leveling in go-nuts + the fanout consumers rather than adding an exclusion, with the reasoning for why suppression is the worse option.

Two things to carry forward:

The verification still has a live step. Everything I confirmed was local — the binary's output now satisfies the collector's conditions, and a control group of already-JSON services proves that pathway produces status:error. But nothing is deployed. After rollout, confirm with a query that can come back empty:

Zero rows means the fix didn't take. Today that query returns nothing across 24h despite 464 ERROR lines existing.

One caveat I'd flag before rollout. message changes shape for these services: today it's the whole raw line, after the fix it's the slog msg only, with the detail moving into @error and other attributes. I checked that no monitor or log-metric in infra depends on this, so no alerting breaks — but any personal saved views, notebooks, or dashboard text queries someone has built against these services' raw log text would need adjusting. That's outside what I could audit from the repos.

PR created.

Trail #94 — https://entire.io/gh/entirehq/mirror-pipeline/trails/94 Branch nodo/cor-1226-... → main, type bug, open, 10 files, +231/−0.

One thing worth knowing about what got created: GitHub PR #238 is not the review surface. Entire opened it automatically as a shadow PR to run CI — its body is just a backlink, and it's marked draft by design. The trail is where the description and review live, so I've left #238 alone rather than marking it ready. Linear had already auto-linked it to COR-1226 via the branch name; I added the trail link alongside it, so the ticket now points at both.

The trail body covers the cause, the two deliberate deviations from the sibling slogutil packages, the before/after verification, and two reviewer notes — that message changes shape for these services (with the audit showing no alerting depends on it), and that [k8s] Application error log surge should be expected to start firing at notice level.

No reviewers requested yet — entire trail update --add-reviewer <login> when you want to put it in front of someone.

Merged and building — but not yet verified, so I'd hold COR-1226 in In Progress a little longer.

Current state:

• main has it — merge commit 3cc4892.
• Deploy run in progress — run 31412242725, started 17:05Z. The pipeline is test → build 8 binaries → staging deploy → per-service prod promote PRs that auto-merge into fleet, then Flux rolls the pods. No manual gate anywhere.
• Not live yet — the 8 services are still 100% info over the last 20 minutes, which is exactly what I'd expect this early.

I've set a watcher on the deploy run; it'll ping me when the run finishes. Once it does I'll give Flux a moment to roll the pods and then run the real check:

That query returns zero today despite 464 ERROR lines existing in 24h, so a non-empty result is genuine evidence and an empty one means the fix didn't take — the distinction the whole ticket turns on. I'll also confirm the warn bucket appears, since that's the much larger volume, and re-check that the message field reshaped the way I predicted.

Worth flagging one thing to keep half an eye on: the 8 services roll independently as their promote PRs merge, so you may see a window where some are reporting real severities and others aren't. That's the rollout, not a partial failure.

In flight, nothing verified yet. Roughly 12 minutes in.

Pipeline progress:

• ✅ Tests, all 9 image builds, all staging deploys
• ✅ Prod promote merged for 7 of 9 — webhook-forwarder, webhook-ingest, worker, core-fanout, trails-fanout, fanout, mirrorctl
• 🔄 meta-fanout promoting, rekicker queued

But no pod is running the new code yet. The tell is unambiguous — staging logs at 17:15:51Z, after its deploy job went green, still read:

Old format, empty severity. That's the expected gap: the deploy/promote jobs only merge a fleet commit pinning the new image tag; Flux then has to reconcile and roll the pods on its own interval. So "promote succeeded" is not "deployed" — which is exactly the kind of green step I don't want to mistake for a result.

One incidental confirmation the pipeline is carrying our commit: a fanout log line shows refs/heads/promote/mirror-pipeline-admin/main-3cc4892 flowing through as a webhook.

My watcher is still polling and will ping me when the workflow finishes. After that I'll wait for the pod roll and then run the real check. The signal I'm looking for, in order:

1. custom.status stops being "" and carries INFO/WARN/ERROR
2. message becomes the bare slog msg instead of the raw timestamped line
3. The status:error query returns non-zero

If the workflow completes and the pods still haven't rolled after a reasonable wait, that points at Flux reconciliation rather than anything in the change, and I'd go look at the fleet side.

Deployed and verified — and the verification caught a second bug.

The main fix works. Measured over 8 minutes once pods rolled:

The status:error query that returned nothing all day now returns real rows. Mid-roll I could see both formats side by side, which rules out anything else having caused it.

The second bug. webhook-forwarder came back 100% info with custom.status: "200" instead of INFO. A slog attribute named status gets lifted into Datadog's reserved status field, where the remapper reads it as severity — and a value it can't parse as one silently degrades the record to info, discarding the OTLP severity.

I nearly asserted this from the one 200 line, but that line was INFO anyway so it proved nothing. Checking properly org-wide: every log carrying an app status of 400/401/403/404/409/500/502/503 over 7 days indexed as info — ~28k lines, on services that emit real status:error elsewhere. That's conclusive.

It mattered because two of webhook-forwarder's three slog.Int("status", ...) sites are ErrorContext on the paths that dead-letter a delivery — the INC-49 failure mode. Those would have stayed invisible to status:error even after this ticket shipped. Fixed in trail #95 (http.status_code / mirror.status), with an AST guard test that fails on any reserved key — mutation-checked, since a passing test proves nothing here.

Two things need your attention:

This isn't our bug alone. Most of those 28k suppressed lines are entire-api and entire-core-*, silently downgrading records since long before this ticket — the very services whose status:error monitors COR-1226 set out to make trustworthy. Their span monitors are fine; log-based severity on them is partially blind today. I haven't touched those repos; it wants its own ticket.

Trail #95 needs review and merge — the main fix is live in prod with the webhook-forwarder gap still open until it lands.

The one needing review:

Trail #95 — https://entire.io/gh/entirehq/mirror-pipeline/trails/95 Stop app status attributes from suppressing log severity

The rest, for reference:

• Trail #94 (merged, live in prod) — https://entire.io/gh/entirehq/mirror-pipeline/trails/94
• COR-1226 — https://linear.app/entirehq/issue/COR-1226/correct-datadog-severity-mapping-for-go-slog-error-logs
• COR-1228 (NATS reconnect noise, backlog) — https://linear.app/entirehq/issue/COR-1228/transient-nats-reconnects-are-logged-at-error-across-mirror-pipeline

COR-1229 — https://linear.app/entirehq/issue/COR-1229/rename-reserved-status-slog-attributes-in-entire-api-and-entiredb Assigned to you, Backlog, linked to COR-1226.

I grepped both repos so it's actionable rather than a pointer: 7 call sites in entire-api across 6 files (internal/httpapi/errors.go:67 is on an error path), and 22 in entiredb across 15 files — five of those are slogutil.Error, in the CI-webhooks dispatch and core ops-token rejection paths. Plus 3 slog.String("msg", ...) in entiredb, same class, colliding with the message body.

Two things I scoped out after checking rather than assuming: source, trace_id, span_id and service are all reserved in Datadog's docs but verified fine on this path — source lands as custom.source with the tag intact, and the trace_id/span_id in both repos' slogctx.go are deliberate and drive trace correlation. Only status and msg need touching.

Worth being clear about the blast radius: unlike mirror-pipeline, both repos have always emitted JSON, so this has been suppressing severity for as long as those call sites have existed. It's independent of COR-1226, not caused by it.

On the review finding — you were right, and I'd noted that exact limitation to myself while writing the test and then didn't act on it. Fixed in 6fc9d11, pushed to trail #95, with a reply thread posted.

The guard now inspects the variadic region of Debug/Info/Warn/Error, their *Context variants, Log, With, and WithGroup. I mutation-tested all three implicit forms rather than assuming coverage:

One deliberate imprecision worth your judgement: it reports reserved literals in that region regardless of key/value parity, because mixing slog.Attr into a variadic list shifts the alternation and position can't be trusted. So a reserved word used as a value would false-positive. I took that trade because the failure being guarded against is invisible while a false positive is a loud one-line fix — but say the word if you'd rather it were precise.

Both commits are on main via merge 9bb7ff9, and deploy run 31414739089 is building. Watching it.

While that runs, here's the check I'll use — chosen so it doesn't depend on a rare event:

webhook-forwarder's error paths (auth reject, 4xx terminate) fired zero times in 90 minutes earlier, so waiting for one to prove the fix would be waiting on luck. Instead the tell is on its high-volume Forwarded line, which fires constantly:

If custom.status still reads "200" after the roll, the collision is still there. That's a check that can come back wrong, on a line that's guaranteed to be present — unlike "no errors appeared, so it must be fixed."

Current state of the two changes:

• Trail #94 — live in prod since 17:18Z, verified: 6 of 8 services now report real warn/error.
• Trail #95 — merged, building. Until it rolls, webhook-forwarder's two dead-lettering error paths are still being suppressed to info.

I'll report once the pods have rolled. Nothing needs you in the meantime.

Recorded on COR-1228.

Nothing needs doing about the alert itself — it's correct, uninteresting, self-recovered, and P3/non-paging.

Two things worth your attention:

Expect it again shortly. The trail #95 deploy is rolling right now, which will restart the same pods and produce the same burst. If you see a second [k8s] Application error log surge in the next few minutes, that's this — not a new problem.

COR-1228 is more valuable than I priced it. I filed it as NATS-restart noise; it's actually deploy noise, so it fires on every mirror-pipeline release. That's frequent enough to erode the monitor's meaning — a signal that cries wolf on every deploy is one people learn to skim, which is roughly how INC-49 went unnoticed for 52 minutes. The fix is small: a fetch interrupted by its own SIGTERM shutdown isn't an error, so gate the ERROR level on "not shutting down" and let genuine fetch failures keep it.

Happy to do that now while the deploy finishes — it's a contained change in the fanout consumers plus go-nuts, and it'd stop the bleeding before people start filtering these out mentally. Say the word.

Meanwhile the watcher is still on run 31414739089, and I'll confirm webhook-forwarder's custom.status flips from "200" to INFO once its pods land.

Second burst fired, as predicted:

Both over the 50/5min threshold, so expect two more P3 notices from the #95 roll. Same signature as the first — single-minute spike coincident with the pod roll, immediate return to baseline. I haven't re-sampled the message text on this one, but the timing and shape match the burst I did verify.

Status summary

Two deploys in 40 minutes produced three false surges. That's the pattern COR-1228 describes, now with live evidence behind it. The fix is contained — gate the ERROR level in the fanout consumers on "not shutting down" so a fetch interrupted by its own SIGTERM stops counting as an error, and leave genuine fetch failures loud.

I'd suggest doing that next, since every future mirror-pipeline deploy trips this until it lands. Want me to?

Deploy run 31414739089 completed successfully, no failed jobs. Severity is working across the fleet:

But chasing the emergency lines turned up something I'd underestimated. Two mirror-worker records came back at emergency — the highest Datadog severity. The cause was custom.status: failed: the donejobs job state landing in the reserved field, where Datadog's remapper reads "failed" and escalates it.

So the reserved-status collision doesn't only suppress severity — it can escalate it too. An slog.Info("Published mirror completion") line was arriving as emergency simply because the job it described had failed. I fixed that call site in #95 on general hygiene grounds, describing it as "benign today"; that was wrong, and understated it.

It's already resolved by #95 — the roll is just mid-flight, with both generations visible on the same hosts:

I've got a short settle window running and will confirm the tail: zero emergency, and nothing but severity words in that attribute.

Two notes while that finishes:

This strengthens COR-1229. I scoped that ticket around severity being suppressed in entire-api and entiredb. The escalation direction matters too — entiredb/internal/entirecore/mirrordone/consumer.go and composite.go log exactly this kind of state string as status. I'll update the ticket once the verification lands.

My question from before is still open — whether to fix the deploy-noise re-leveling (COR-1228) now. Nothing has been actioned on that.

Both halves are ready.

Trail #96 — https://entire.io/gh/entirehq/mirror-pipeline/trails/96 All 8 call sites wired, 30 packages green, lint clean.

The approach turned out to be generalising an existing convention rather than inventing one. Those loops already log ErrStreamNotFound at warn with the comment "so operator alerts on ERROR aren't tripped by normal startup ordering" — exactly this reasoning, applied to one error. natslog.Err extends it to the rest of the class.

Three deliberate constraints:

• No control-flow change anywhere. Every site keeps its return-and-resubscribe or continue-with-backoff behaviour. Only severity changes, so this can't affect how consumers recover or introduce a hot loop.
• meta-fanout got a real bug fix in passing — its lifecycle loop never had the shutdown guard its siblings carry, which is why Lifecycle fetch error appeared in the bursts. A clean teardown was surfacing as a fault.
• context.DeadlineExceeded and ErrStreamNotFound stay at error, with tests, so a wedged server is as loud as it was.

One correction on my own method: my first mutation check used if false, which failed to compile on an unused import — so it "failed" for the wrong reason and proved nothing. Redone as && false, which gave a real signal: the four transient cases fail by name.

One thing needs you. go.mod pins go-nuts at the merged main commit as a pseudo-version:

That builds and ships fine, but go-nuts tags proper releases and mirror-pipeline pinned v0.9.0 before this. When you tag v0.10.0, the swap is one line — I'd rather you pick the version than have me cut a release.

The verification is written into the trail as a two-sided check: after the next deploy, the roll should produce status:warn where it produced status:error, and the surge monitor should stay quiet. An empty warn result would mean the lines vanished rather than moved — which is why I check both sides rather than just confirming the alert didn't fire.

Both done.

go-nuts version — already latest. origin/main is d46cef38, and mirror-pipeline's go.mod pins exactly that (v0.9.1-0.20260810180711-d46cef38c172, the merge commit). Nothing to bump, and no release needed for it to ship.

The review finding — correct, fixed in ac72878, pushed to trail #96 with a reply thread.

You were right about the cause, and the reason I missed it is worth stating plainly: I built the call-site list from the log patterns Datadog surfaced during the bursts. That consumer's subscribe failure hadn't fired in my sample window, so it never showed up. Enumerating PullSubscribe/.Fetch( from the source gives nine sites — I worked from eight. Scoping a fix by observed symptoms rather than by the code, which is the same trap as trusting a filter that matches nothing.

So rather than just patch the ninth, there's now a test that fails on any error-level log naming a subscribe or fetch. Mutation-checked against your exact finding. It immediately found two more of the same class:

• cmd/worker/advisory.go:144 — advisory backstop subscriber error, will retry
• pkg/webhookdlq/advisory.go:265 — the webhook-DLQ sibling

Both logged a retrying subscriber failure at error. Migrated, with a new natslog.ErrContext since they carry a context and a backoff attribute.

Two things I'd want a reviewer to push back on if they disagree:

The guard is a text heuristic, deliberately. A structural check — "error log inside a function that calls PullSubscribe" — would have missed both advisory backstops, because they call a helper rather than subscribing directly. The weaker-looking check has the better coverage here. Its cost is a false positive on cmd/rekicker's permission-denied log, which lists the NATS grants an operator must restore (including the verb SUBSCRIBE) and is a genuine permanent fault; a message-length bound excludes it, since real labels are terse and that one runs a couple hundred characters.

I widened scope past your finding by migrating those two advisory sites. Same defect class and same fix, but it's more than you flagged — say the word if you'd rather they were split out.

30 packages pass, lint clean. Trail #96 is ready for another look.