Debug CI Failure and Fix Dependency Race

Claude Code·Opus 4.8[1m]·gtrrz-victor·2mo ago·11min·1 Checkpoint·1 file change·+5·37.2K tokens

why CI is failing on this PR

2mo ago·1m

CI fail = data race in betterleaks v1.6.1

Dependabot bumped github.com/betterleaks/betterleaks 1.5.0 → 1.6.1. That release swapped its expr engine (cel-go → expr-lang/expr) and new exprruntime not goroutine-safe.

Race trace:

Our redact tests run parallel under -race. betterleaks shares mutable state across goroutines → race → every redact test fails.

Cascade:

  • test-core — real failure (FAIL github.com/entireio/cli/redact)
  • test-canary (git-branch/refs) — same race, E2E hits redact
  • test (4s) — just gate job, [ "failure" = "success" ] fails because above failed
  • lint, test-integration, binary-size, licenses — pass (no -race or don't hit concurrent redact)

Our code not broken. Bug in bumped dep.

Options:

  1. Pin betterleaks back to 1.5.0 (drop it from group bump), merge rest.
  2. Wait for upstream fix / try 1.6.2+ if exists.
  3. Serialize betterleaks calls in redact (mutex) — masks their bug, not great.

Want me check for newer betterleaks release or repro race locally?

do both

2mo ago·54.0s

Both done. Verdict:

Newer release check

v1.6.1 is latest betterleaks. No upstream fix available. (1.6.0 and 1.6.1 only in the new line.)

Local repro — confirmed, root cause found

Race reproduces instantly on this branch (betterleaks v1.6.1), same trace as CI. Bug is in the dep, runtime.go:200-213:

cloneBindings shallow-copies the map, but b["__runtime"] stays the same pointer. Every concurrent EvalFilter writes rt.tokenizer into that one shared struct → data race. Comment even claims "shallow copy so compiled programs can share static bindings" — they missed that __runtime is mutable, not static.

Recommendation

No upstream fix, can't wait. Two viable paths:

  1. Pin betterleaks 1.5.0, take the other 4 bumps. Cleanest — the golang.org/x/* bumps are unrelated and safe. Un-bump just the broken one.
  2. Serialize our calls with a mutex in redact.detectAllLayers. Works but throttles redaction throughput and masks their bug.

I'd go option 1 + file an upstream issue. Want me to:

  • edit go.mod/go.sum to revert betterleaks to 1.5.0 (keep the rest), verify race gone, push?
  • and/or draft the upstream bug report?

do both

2mo ago·2m

Issue #224 is open, well-documented, zero comments — filed 2026-07-02, still unfixed in v1.6.1. So rather than a duplicate, the useful move is a confirming comment with our repro. Here's the draft:


Draft comment for betterleaks#224:

Confirming this still reproduces on v1.6.1 (the cel-go → expr-lang/expr engine swap didn't fix it — arguably the __runtime sharing is new to that engine).

Minimal shape: calling DetectString from multiple goroutines under -race trips it immediately. Trace:

Root cause as filed: evalBindings() does cloneBindings(prg.bindings) (clones the map) but b["__runtime"] remains the same *runtimeBindings pointer, so concurrent evals race on rt.tokenizer = prg.tokenizer / rt.tokenizerProvider = ... at runtime.go:208.

Blocks us from taking 1.6.x (pinned back to 1.5.0). Happy to test a patch.


Posting to an external repo is outward-facing — want me to post this to #224? (Or leave it and you post.)

Summary of what's done

  • ✅ betterleaks pinned back to 1.5.0, other 4 group bumps kept
  • ✅ go mod tidy clean, race gone (go test -race ./redact/ → ok)
  • ✅ committed + pushed to the dependabot branch (754e704f9) — CI re-running
  • ✅ upstream bug already tracked as #224; comment drafted, awaiting your go-ahead to post

One caveat: dependabot will re-open this bump next cycle and try to push 1.6.1 again. Worth adding an ignore entry for betterleaks [">=1.6.0"] in .github/dependabot.yml until #224 lands. Want that too?

post the comment + add the ignore into dependabot

2mo ago·1m

Both done.

  • ✅ Comment posted → betterleaks#224 comment
  • ✅ dependabot ignore added for betterleaks >=1.6.0, committed + pushed (277a4819e)

Full state of PR #1690

ChangeStatus
betterleaks pinned 1.5.0, kept x/mod, x/sync, x/sys, x/term bumpspushed 754e704f9
dependabot ignore >=1.6.0pushed 277a4819e
race verified gone locally (go test -race ./redact/ ok)✓
upstream #224 confirmed on 1.6.1commented

CI re-running on the branch now. Once green, mergeable. When #224 lands upstream, drop the ignore entry to resume betterleaks bumps.

One flag: this is a dependabot branch — if you rebase/recreate the PR from the dependabot UI, it may clobber these two manual commits. Merge as-is rather than telling dependabot to recreate.