Bottom line

It's not random flakiness — the same 4 agents fail nearly every run for distinct, diagnosable reasons, and because they share one matrix with the reliable agents, main goes red on essentially every merge. The Slack alert is now pure noise, so a real regression in claude-code would be invisible. Fix the signal architecture first, then the individual tests.

What the data actually shows (last ~20 runs on main)

Every run = failure or cancelled. Per-job, it's consistent:

• Always green: claude-code, opencode, gemini-cli, roger-roger, e2e-windows
• Always red: cursor-cli, copilot-cli, codex, factoryai-droid

And the red ones fail for different reasons — this matters:

So: a long tail of specific failing tests + one quota problem + one genuinely-broken agent + concurrency cancellations. There is zero retry anywhere (gotestsum ... -count=1, no --rerun-fails).

Fixes, ranked by leverage

1. Split the matrix into tiers (biggest win). Reliable agents (claude-code, opencode, gemini-cli, roger-roger/vogon canary) stay as the gating + Slack-alerting set. Move cursor/copilot/codex/factory to continue-on-error: true or a separate non-paging workflow until fixed. This instantly makes green achievable and the alert meaningful again — without losing any coverage.

2. Add automatic reruns: gotestsum --rerun-fails=2 --packages=./e2e/tests in mise-tasks/test/e2e/_default. Kills the transient single-test tail (codex/copilot). Won't fix cursor's quota or factory's 14 — those are real.

3. cursor-cli: "Increase limits" is an account/quota ceiling, not code. Add a cursor-cli entry to E2E_CONCURRENT_TEST_LIMIT (the workflow already special-cases gemini=6, factory=1) and/or raise the plan. No amount of retry fixes a quota wall.

4. factoryai-droid: 14 failures is a real broken integration — quarantine it out of the gating set with a tracking issue rather than letting it page every merge.

5. Quarantine the specific chronic tests (codex resume, copilot shadow-branch) with t.Skip("ENT-XXX: ...") + issues, using the existing /e2e:triage-ci skill against artifacts to confirm flaky-vs-real first.

6. Concurrency: post-merge cancel-in-progress: true turns every rapid merge into a cancelled run. Switch to a queue (cancel-in-progress: false) so you stop reading cancellations as failures.

#1 + #2 alone would make main green on most merges and restore a trustworthy alert.

Want me to implement #1 and #2 (tiered matrix + rerun-fails)? Those are low-risk and high-leverage. I'd leave #3–#6 as follow-up issues since they need per-test triage and an account/quota decision that's yours to make.

Done. All three changes made and validated (YAML parses, shell logic unit-tested across cases, sh syntax clean).

What changed

1. Tiered Slack alerts (.github/workflows/e2e.yml) — the notify-slack job now classifies failed jobs:

• RED (#d50200, :red_circle: "E2E Tests Failed"): any of claude-code, opencode, gemini-cli, roger-roger, or e2e-windows fails → real regression.
• YELLOW (#daa038, :large_yellow_circle: "flaky-agent failures"): only cursor-cli/copilot-cli/codex/factoryai-droid fail.
• A mixed failure is RED but lists both tiers in the body, so you never lose sight of which flaky agents also broke. Switched from plain blocks to attachments so the colored bar actually renders. The job still fires on any failure — no visibility regression.

2. Automatic reruns (mise-tasks/test/e2e/_default) — gotestsum --rerun-fails=1 --packages=./e2e/tests. One retry of only the failed tests; a test that passes on retry reports green. Required moving the package from a positional arg to --packages. The default --rerun-fails-max-failures=10 means a broadly-broken run (e.g. factoryai-droid's 14) is reported as-is, not masked. Windows inherits this (same task). Canary/roger-roger untouched — deterministic agents stay strict so real bugs aren't hidden.

3. cursor-cli throttle (.github/workflows/e2e.yml) — added cursor-cli → 2 to E2E_CONCURRENT_TEST_LIMIT (its failures were Increase limits for faster responses = account rate-limit). Picked 2 as a conservative throttle alongside gemini=6, factory=1; bump it if cursor still rate-limits.

Notes

• Not committed (per your workflow).
• One residual noise source I left alone: concurrency: cancel-in-progress: true turns rapid back-to-back merges into cancelled runs. Not a failure, but it's why you also see grey. Say the word if you want it switched to a queue.
• cursor's quota and factoryai-droid's 14 failures are real and won't be fixed by reruns — they're now correctly YELLOW instead of reddening main. Worth separate triage tickets.

Done. Three separate commits, working tree clean:

• c7e3126727 — rerun failed tests once (mise-tasks/test/e2e/_default)
• 4eb09acae3 — throttle cursor-cli concurrency (e2e.yml)
• 150bbbb814 — tiered red/yellow Slack alert (e2e.yml)

Not pushed — yours to push.

Bottom line: push is blocked — the GitHub App token behind the entire:// remote lacks workflows permission, so it refuses any change to .github/workflows/. This isn't something I can work around from here.

The commits are ready locally. You'll need to push with your own credentials (which have workflows scope). Run this in the session:

Once it lands, I'll create the PR. Or if you'd rather, push + open the PR yourself in one go and I'll skip it.

One thing to confirm before the PR: what's the base — main, right?

• Fix broken tests: Triage the small per-agent failures with /e2e:triage-ci against artifacts — codex's TestCodexResumeRestoredSessionWithSanitizedCompactedHistory ("session_meta id not found") and copilot-cli's shadow-branch leak ("should be empty, but was [entire/...]") — classify each as real bug vs flaky and fix or t.Skip("ENT-XXX").

• factoryai-droid: Decide if it's salvageable — 14 failures is broad, so check whether it's one systemic integration issue or genuine agent non-determinism, and if not fixable soon, drop it from the gating matrix (it's already YELLOW now) with a tracking issue.

• cursor quota: Either accept the concurrency=2 throttle or raise the Cursor account plan limit — reruns won't fix a hard rate-limit wall.

• Concurrency cancellations: Optionally switch cancel-in-progress: true → a queue so rapid merges stop showing as grey cancelled runs.

• This PR: Push it yourself (workflows-scope) and I'll open the PR — then watch the next main run to confirm green/yellow actually happens.

Branched off latest main (my workflow fixes live on the other branch — irrelevant here). Now invoking the triage skill on the two always-failing tests.

I have everything I need. Here's the triage.

Triage findings — 2 chronically-failing tests (CI run 28434323127)

1. TestCodexResumeRestoredSessionWithSanitizedCompactedHistory (codex) — flaky (test-bug)

Re-run signal: consistent failure, but root cause is in e2e/, not the CLI.

Evidence:

• Fails at e2e/tests/codex_resume_test.go:82 in the test's own helper readCodexSessionID, which scrapes the session ID out of the rollout file with a brittle regex: ^\{"timestamp":".*","type":"session_meta","payload":\{"id":"…
• That regex assumes a fixed field order. Codex v0.142.4 (was 0.116.0 when the regex was written) reordered the session_meta line, so the regex matches nothing → "[] should have 2 item(s), but has 0".
• The CLI itself parses it fine — entire.log shows the SessionStart hook read the rollout and resolved session_id 019f17e1-…. The CLI uses robust JSON decoding (agent/codex/transcript.go: unmarshal line → type=="session_meta" → unmarshal payload), which doesn't care about field order.

Proposed fix:

• File: e2e/tests/codex_resume_test.go
• Change: replace the regex in readCodexSessionID with line-by-line JSON parsing (decode each line, find type=="session_meta", read payload.id) — mirroring the CLI. (Or read the ID straight from the rollout filename rollout-<ts>-<id>.jsonl.) Either is robust to future Codex format drift.

2. TestSubagentCommitFlow (copilot-cli) — real-bug (CLI)

Re-run signal: deterministic; a wait won't help — the branch will never be cleaned up in this state.

Evidence (from entire.log):

• Subagent task checkpoint created on shadow branch entire/32fbd6e-e3b0c4 under a subagent session REDACTED (a Bedrock tool-use id).
• At user-commit, the main session condensed correctly, but the subagent session was still phase: active, so post-commit logged: preserving shadow branch (active session exists) and skipped cleanup → entire/32fbd6e-e3b0c4 leaks → test fails at line 45.
• Related copilot-only defect in the same flow: the task checkpoint was saved with empty tool_use_id/subagent_type, producing a malformed metadata path …/tasks//checkpoint.json ("path contains empty segment") that got skipped with a WARN.
• claude-code passes this same test → copilot-specific.

Root cause analysis:

• Component: agent integration (subagent/task session lifecycle) + strategy post-commit cleanup.
• Suspected location: Copilot CLI integration — it registers a subagent tool-use as a session but never delivers a terminating (Stop/SubagentStop) event, so the session sticks in active; the strategy then correctly-but-unhelpfully refuses to delete a shadow branch with a live session.
• Proposed fix: end the subagent session when the task completes (and populate tool_use_id/subagent_type so the metadata path is valid). Needs a real code change in the copilot integration — not a test tweak.

Summary

So: one is a quick test fix, one is a genuine product bug worth a tracking ticket.

Re: that PR comment (#1577 r3498333535)

It's a correct bug report from the Copilot review bot about my rerun change — and you should act on it.

In plain terms: I added --rerun-fails=1 to the shared _default e2e task and wrote a comment saying "don't add reruns to deterministic agents like roger-roger." But the workflow runs roger-roger through that same shared task (mise run test:e2e --agent roger-roger …), so roger-roger now gets retried too — exactly what I said it shouldn't. A real roger-roger regression could be masked by a retry. My own change contradicts my own stated intent.

Fix (one line, on the #1577 branch): point the roger-roger leg at the dedicated no-rerun task — mise run test:e2e:roger-roger TestExternalAgent instead of mise run test:e2e --agent roger-roger TestExternalAgent. That task has no --rerun-fails, so the deterministic agent stays strict.

Want me to:

1. Apply the codex test fix here on this triage branch, and
2. Hop to the #1577 branch and fix the roger-roger rerun leak the bot flagged?

(The copilot real-bug I'd leave as a ticket — it's a deeper integration fix, separate PR.)

All three tasks done.

Bottom line

Two commits added to PR #1577 (pushed), and PR #1578 opened off main for the copilot real-bug.

PR #1577 (magical-yawning-matsumoto) — 2 new commits pushed

• 1386ab4f0 — codex test fix: readCodexSessionID now parses the rollout JSON line-by-line instead of an order-sensitive regex that Codex 0.142.x broke. Verified with go vet -tags=e2e.
• e20f01d0b — addresses the review comment (r3498333535): roger-roger now runs via its dedicated no-rerun task, so the deterministic agent can't be masked by --rerun-fails.

PR #1578 — copilot TestSubagentCommitFlow real-bug

https://github.com/entireio/cli/pull/1578 (branch off origin/main)

• Root cause: Copilot fires per-turn lifecycle hooks for subagent turns using the Task tool-use id (toolu_…) as the sessionId → a phantom session that never ends → pins its shadow branch open after commit → the leaked entire/<hash> branch the test caught.
• Fix: drop session-lifecycle hooks when the sessionId has the toolu_ prefix (real Copilot ids are UUIDs); subagent-stop untouched so task checkpoints still record.
• Verified: new copilot unit tests pass; full mise run lint clean.
• Caveat I flagged in the PR: unit-tested, but a real copilot-cli E2E run of TestSubagentCommitFlow should confirm end-to-end before merge — I didn't run it (real API cost, and per repo rules I don't run E2E proactively).

Notes

• Both pushes went to github.com:entireio/cli.git directly and succeeded (no workflows-permission wall this time, since #1578 touches no workflow files and #1577's push went through).
• One related copilot defect is still open as out-of-scope: the task checkpoint saves with an empty tool_use_id (malformed tasks//checkpoint.json, WARN-only) — worth a separate ticket, doesn't affect the test.

Want me to reply to the bot's review comment on #1577 marking it resolved, or run the copilot E2E to fully validate #1578?

Bot was right — pushed the fix to #1577.

What it caught (valid, Medium)

E2E_CONCURRENT_TEST_LIMIT=2 for cursor-cli was a no-op. AcquireSlot only throttles agents that called RegisterGate; cursor-cli never did, so it ran at full parallelism and kept hitting "Increase limits for faster responses." My earlier change set the env var but never wired the gate.

Fix (commit 13a04b964, pushed to #1577)

• Added RegisterGate("cursor-cli", 2) in e2e/agents/cursor_cli.go — now the throttle actually applies.

One more thing the bot didn't flag

factoryai-droid has the identical latent no-op — the workflow sets E2E_CONCURRENT_TEST_LIMIT=1 for it, but it also never calls RegisterGate, so factory has been running ungated this whole time. Only codex/gemini/copilot actually gate today.

I left factory alone on purpose: its 14 failures aren't concurrency-related (per the triage), and "fixing" it to truly serialize would make that job ~Nx slower for no current benefit. Your call:

• Leave it (status quo, ungated), or
• Register its gate so =1 serializes it as originally intended.

Want me to register the factory gate too, or leave it?

Done — pushed to #1577 (489e82526).

• RegisterGate("factoryai-droid", 1) added in e2e/agents/droid.go, so the workflow's E2E_CONCURRENT_TEST_LIMIT=1 now actually serializes it.
• Both gates (cursor-cli=2, factory=1) are now wired; all gated agents respect their limits.

#1577 now has 6 commits total (3 original + codex test, roger-roger rerun, cursor gate, factory gate). All gate/throttle env vars are now honored.

Root cause found and fixed on the branch (committed, not pushed).

Why factoryai-droid is a mess — solved

Every factory failure across all 3 runs is the same startup auth death, and it's a test-harness bug, not flakiness, not a CLI bug, and not rate-limiting:

• Interactive StartSession launched bare droid with no --model → droid falls back to Factory-hosted models → validates FACTORY_API_KEY at startup → "appears to be invalid" → exits status 1 before the > prompt.
• Headless RunPrompt passes --model custom:<haiku> (BYOK via ANTHROPIC_API_KEY) → works.

Evidence that nails it:

• The failing set is stable: the same 7 tests every run, all StartSession-based; the lone extra (TestAgentAmendsCommit, a RunPrompt test) failed once on a genuine transient.
• RunPrompt has FACTORY_API_KEY set too and still passes — so the differentiator is the missing --model flag, not the key.

Fix (commit 7fe116c0c)

Pass the BYOK --model defaultDroidModel to the interactive tmux launch, mirroring RunPrompt. One line + comment.

Two things worth knowing

• The concurrency gate does NOT fix factory. Its failures aren't rate-limiting — they're structural auth-at-startup. The gate is still fine as hygiene (and it's already in #1577), but don't expect it to green factory.
• Validation gap: I can't run real droid E2E here (binary + key, and the no-proactive-E2E rule). The fix is strongly evidence-backed, but confirm with a real factoryai-droid E2E run before trusting it. Small residual assumption: interactive droid accepts --model (near-certain, since droid exec --model works).

Branch state

investigate/factory-droid-e2e = factory gate (cherry-picked from #1577) + the droid --model fix. Not pushed.

Heads-up: the cherry-picked gate commit duplicates #1577. When I open the PR I'd recommend dropping it here (rebuild the branch off main with just the --model fix) so we don't carry a dup — or leave it and let git dedupe on merge.

Want me to push and open the PR (gate dropped), and/or kick off a factory E2E run to validate?