Fix Checkpoint Token Double Count Bug

I found the other session in this worktree (7ee685c3, last active 13:29) and caught up on it here. If you'd rather reopen that exact conversation, run claude --resume 7ee685c3-5ca9-4c70-8693-3e4af96e02eb.

Where it stopped: it was designing entire checkpoint delete. No code has been written. The branch is still at main (89c261608) with no commits. The design spec is in docs/superpowers/specs/2026-10-02-checkpoint-delete-design.md, a gitignored folder, so it's only on this machine.

Already decided (Oct 5):

• The CLI command, the backend change and the frontend fix ship together.
• On entire.io, deletes are handled when the server receives the ref delete, and the CLI gets --dry-run.
• If more than one remote holds the checkpoint, you pick which ones in an interactive list. Without a prompt, --force deletes from all of them.

Its last finding: tokens are already double counted today, separate from the delete work. After a partial commit or session adopt, the next checkpoint counts the session's tokens from the start again, including tokens earlier checkpoints already counted (manual_commit_hooks.go:3881, session_adopt.go:474). The per-agent charts on entire.io also credit a checkpoint's whole token total to one agent, even when it holds sessions from several.

Decisions it was waiting on from you:

1. Token double-count bug: fix it in its own PR first, or include it in the delete work? It recommended a separate PR first.
2. Per-agent charts: count tokens per session before delete ships, or accept that a whole checkpoint is credited to one agent?
3. Deleted turns: after a delete, those turns still show in the session transcript but no longer count toward tokens. Is that acceptable, or should they be hidden too? Hiding them is much more work.
4. Old branch-stored (v1) checkpoints: refuse and point to entire doctor migrate, or remove them in a new commit while warning that branch history still has them? It recommended refusing.
5. Checkpoint held by an active session: refuse unless --force, or quietly clear the session's state? It recommended refusing unless --force.

Answer these and I'll finish the spec, have a reviewer check it, then start on the CLI command.

The double-count bug is real. I reproduced it with an integration test: after a partial commit, the next checkpoint stores the earlier turns' tokens again.

The bug

Where: when a commit leaves some agent-changed files uncommitted, the CLI carries those files forward and resets the session's transcript offset to 0 (strategy/manual_commit_hooks.go:3881). session adopt does the same (session_adopt.go:474). The offset exists so each checkpoint keeps the full transcript. But condensation also uses that offset to work out which tokens belong to the checkpoint (strategy/manual_commit_condensation.go:1544).

What gets double counted: the first checkpoint after a partial commit stores every token from the start of the session, not just the new turns. Repro:

Without the partial commit, checkpoint 2 stores 7, which is correct. The extra amount is every token up to the previous checkpoint, so it grows with session length. It repeats after each partial commit.

Which agents: Claude Code, Factory Droid, OpenCode, Pi, Codex, Copilot CLI, and external agents that respect the offset. Cursor and Antigravity aren't affected because they get token counts from hooks. Subagent tokens aren't affected.

Local entire status is correct. It uses the session's running total, not a sum of checkpoints.

Effect on entire.io

Every number that adds up per-checkpoint tokens is inflated:

• Session pages: the session's total tokens.
• Commit page: the "b and c" commit shows 107.
• Repo overview: the summary, activity over time, contributors, per-agent breakdown, and agent-activity charts.
• Home page: activity and contribution totals.
• Recap: per-agent totals.

Commit and line counts are unaffected because they don't use tokens.

I read this from an entire-api checkout dated Sep 17. Its remote now returns "Repository not found", so I couldn't check the current server code. Where does entire-api live now?

Fix: keep a separate offset just for token counting, so carry-forward and adopt can still store the full transcript. I'd do it as its own PR off main, separate from delete. Checkpoints already written stay inflated unless the server recalculates them, which would be a separate decision.

3. "Removed" needs one more decision

Every checkpoint stores the session's full transcript up to that point. The test confirmed checkpoint 2 contains turn 1's prompt, even without a partial commit. So deleting checkpoint 1 alone can't remove turn 1: checkpoint 2 still has it. Two ways to make "no transcripts left" true:

• A. Delete the whole session (recommended). Deleting a checkpoint also deletes every other checkpoint holding the same session, after confirmation. It's simple and fully removes the content. The catch: a checkpoint shared with another session loses that session's data too, so the prompt has to list those sessions.
• B. Rewrite the later checkpoints to cut the deleted turns out. It keeps the rest of the session, but each later checkpoint must be rewritten and force-pushed, and the token and line credit has to be recalculated. It's much more work and more fragile.

4 and 5 recorded

• v1 checkpoints: deleted with a new commit that removes them from the branch. The prompt says the content stays in the branch's history on the remote. The server will also need to treat that removal as a delete.
• Active session: refuse unless --force.

The spec at docs/superpowers/specs/2026-10-02-checkpoint-delete-design.md has your decisions and the repro numbers. The repro test is in my scratchpad, outside the repo, and can become the regression test for the token fix.

Which do you want for 3, A or B? And should I start the token fix as its own PR?

Your question about session attach has a two-part answer:

• No, the token fix alone doesn't cover attach. session attach counts the whole transcript from the start every time (attach.go:332), and it never records how far it counted. So it double counts in two ways:

  • If the session already has checkpoints but its saved state no longer points to one, the attach checkpoint counts those earlier tokens again. A partial commit clears that pointer, so this happens naturally.
  • If the session keeps running after the attach, the next hook-made checkpoint re-counts what the attach already counted.

  I've included an attach fix in the same PR.

• The multi-worktree linking problem is separate. That's the "commits not captured, have to run session attach" issue from PR #2531, and this PR doesn't touch it.

The plan:

• Separate offset for tokens. Add a second transcript offset that only tracks how far tokens have been counted. It moves forward at every checkpoint and never resets. The partial-commit path keeps resetting the existing offset to 0, so the next checkpoint still shows the full conversation, but tokens are counted from the new offset.
• Attach counts from that offset and records it afterwards.
• Adopt no longer resets it.
• Older session state that lacks the new field falls back to the current offset. Sessions already mid-partial-commit will double count one more time; after that it's fixed.
• Already-inflated numbers on entire.io stay as they are. Recalculating them on the server is a follow-up.

The worktree is .worktrees/token-scope, on branch peyton/checkpoint-token-scope off latest main. The plan is in docs/superpowers/plans/2026-10-05-checkpoint-token-scope.md in that worktree; that folder is gitignored, so it stays local.

A reviewer agent is checking the plan against the code now. I'll bring you its findings and the revised plan before writing any code.

One question for you while it runs: should adopt count tokens the source session never got into a checkpoint? My plan says yes: they get counted once, in the new location. Subagent tokens currently work the other way, counting only what happens after the adopt.

The reviewer agreed with the approach and found gaps, which are now fixed in the plan. I spot-checked two of its claims against the code: review-session tokens and the hook-token fallback.

Changes to the plan from the review:

• Missed tokens after a mid-turn commit. When an agent commits mid-turn, the tokens it uses after the commit for the rest of that turn are never counted anywhere. That's an existing under-count. The new token offset will stay behind at that point, so the next checkpoint counts them. For agents that commit mid-turn, like Codex, totals will go up a little after the fix; that's the correction.
• Attach. Attach changes in three ways:
  • It counts only from the token offset, and moves only that offset forward afterwards. What the next checkpoint's transcript shows stays as it is.
  • It clears the pending per-checkpoint token total and resets the subagent baseline. Otherwise Cursor's hook-reported tokens and subagent tokens get counted twice.
  • It takes the transcript position from the bytes it already read, so lines written in between aren't skipped.
• Review sessions. Their token counting (review/manifest.go:442) moves to the new offset too.
• Added to out of scope: snapshot checkpoints never advance the offset, and a deleted-and-recreated session state restarts at 0.
• New tests: two partial commits in a row adding up exactly; Codex mid-turn tail; Copilot after a partial commit; attach with Cursor and with subagents; review tokens; migration of older session state.

One decision for you: adopt. I asked this last time; it's the last open question in the plan:

• A (recommended): tokens the source session never got into a checkpoint are counted once, in the adopted location. Subagent tokens change to match, which means removing the subagent reset adopt does today.
• B: only count tokens used after the adopt, matching how subagent tokens and prompt counts work today. Any source-side tokens that never reached a checkpoint are lost.

Which one for adopt, A or B? Once you pick, I'll start the fix with the failing partial-commit test first.

Here's an example of what A means. A session does some work in worktree X, then moves to worktree Y, and you run session adopt there:

1. Turns 1–2 happen in X. You commit in X, which creates checkpoint C1 with the tokens for turns 1–2.
2. Turn 3 happens in X, but nothing is committed. Turn 3's tokens aren't in any checkpoint yet.
3. You adopt the session into Y.
4. Turn 4 happens in Y. You commit, which creates checkpoint C2.

What C2 counts:

"Never got into a checkpoint" means turn 3: it happened before the adopt but no commit captured it. With A, each turn is counted exactly once. With B, turn 3 is lost.

The same applies when the session never committed in X at all. A counts turns 1–3 in C2; B drops them.

Should I go with A?

I've made the token fix and the tests I added pass. The full mise run check (format, lint, all CI tests) is still running in the background; I'll report the result when it finishes. Nothing is committed yet.

What changed (.worktrees/token-scope, branch peyton/checkpoint-token-scope, 8 files):

• New offset for tokens. Session state gets a second offset, TokenTranscriptStart, that only moves forward when a checkpoint stores tokens. Token counting uses it in normal checkpoints, the live-transcript path, and review sessions. After a partial commit the next checkpoint still shows the whole conversation, but counts only new tokens.
• Mid-turn commits. The tokens an agent uses after committing mid-turn now land in the next checkpoint. Before, they were never counted anywhere.
• Attach counts only tokens no earlier checkpoint has counted. It then records how far it counted and clears the pending per-checkpoint total, so the next checkpoint doesn't count them again.
• Adopt (your option A) keeps the source session's token position, pending total and subagent baseline. This undoes the July subagent reset (d39ecfdd7) and your June change that cleared the pending total (cf4e18d12), so the PR needs to say that.
• Older session state without the new field falls back to the existing offset. A session already in the middle of a partial commit double counts one more time; after that it's fixed.

Tests:

• Partial commits: three checkpoints store 100 / 7 / 3, where today's code stores 107 for the second one.
• Attach after a partial commit: the attach checkpoint stores 7 and the next normal checkpoint stores 3.
• Mid-turn tail: the next checkpoint stores 12. I confirmed it fails (7) if the token offset is moved at turn end.
• Unit tests: loading older state, and copies of a session state not sharing the new field.
• Adopt: two existing tests were rewritten for option A.

Two things outside this fix:

• Integration-test hooks use your installed entire. The git hooks in integration-test repos run whatever entire is first on PATH. Any test whose commands trigger them runs your installed nightly, not the code under test. I pinned PATH in my attach test only.
• Pending per-turn totals count from the start. In the simulated tests, the pending per-turn total is counted from transcript line 0 (107 after two turns), apparently because the simulated prompt hook doesn't record a transcript position. I haven't checked whether real sessions do the same. It only matters for agents that report no tokens in their transcript, like Cursor.

When the check is green I'll commit and show you the PR description. I'll push and open the PR only once you say go.

The end-to-end run confirms the fix. I ran the same script with the fixed build and with a build of current main. It uses real git hooks and the real Claude Code hook commands, with hand-written transcripts of known token counts and no paid agent calls.

Output tokens stored per checkpoint:

Main gets it wrong in more ways than the integration tests showed:

• A4: committing leftover files with no new turn re-counted the whole session so far (110).
• C3: the checkpoint after an attach re-counted everything again (63).

Staging: I don't think it's needed. The server just adds up the per-checkpoint numbers, and that code doesn't change. A staging run would mean logging in to staging and pushing a test repo there. If you want to see the entire.io session totals come out right (167 instead of 1,016 for these three sessions), I can do that.

mise run check: formatting and lint passed; the test suite is still running. I'll report when it finishes, then commit and show you the PR description before pushing.

The script is in my scratchpad (e2e/scenario.sh), along with the two scratch repos it created.

API Error: Connection lost mid-response. The response above may be incomplete.

Not ready yet. The PR is ready to open once mise run check passes; the re-run is in progress and takes about 11 minutes.

The first run's failures look like load from running the end-to-end test and integration tests on the same machine at the same time:

• Every failure was a test whose helper script got killed after its 5–30 second timeout. None of them touch token code.
• A sample of the failed tests (the external-agent ones) passes on its own.

So I'm re-running the full check with nothing else running.

When it's green: I'll commit and push peyton/checkpoint-token-scope and open the PR. The change is 11 files, +361/−56; about half of that is tests.

PR description draft (in my scratchpad as pr-body.md):

• The four problems: partial commits, attach, adopt, and lost mid-turn tails.
• The fix: a second offset that only tracks tokens.
• That the adopt part reverses d39ecfdd7 and your cf4e18d12.
• The test and end-to-end results, including the before/after table.
• Follow-ups not in the PR: checkpoints already written stay inflated, snapshot checkpoints, and re-created session state.

Should I open it as soon as the check passes, or do you want to read the description first?

I'll open it as soon as the check passes. It's still running; I'll report back when it finishes.

PR #2674 is open: https://github.com/entireio/cli/pull/2674. The full check (format, lint, all CI tests) passed on the re-run with nothing else running.

• Commit: 4ea92d808 on peyton/checkpoint-token-scope, 11 files, +361/−56.
• Description: the draft you saw: the four problems, the separate token offset, that it reverses d39ecfdd7 and your cf4e18d12 for adopt, the before/after table, and the follow-ups.
• Still running: CI and the trail review (trail 1495). I'm watching both and will fix any real findings and resolve them until both are clear. I won't merge.

Trail 1495 is still reviewing. The "0 findings" from the earlier poll came before the review had finished, so it doesn't count. I've restarted the poll with a correct status check, and CI is still running too. I'll report findings and CI results when they come in.

Trail 1495 has two high findings, both valid. The fix is ready and being checked, but not committed or pushed yet.

The findings: both are about the two places that move the displayed transcript window at turn end:

• the turn-end advance after a mid-turn commit (manual_commit_hooks.go:3489)
• the deferred version of it for agents like Antigravity (manual_commit_condensation.go:1806)

If a session's token offset hasn't been set yet, it falls back to the displayed offset. It would then move forward along with it, and the turn's tail tokens would be skipped. State loaded from disk always has the offset filled in, so I don't think this happens today. But that depends on every code path loading state that way, so it's worth closing.

The fix:

• A new helper, AdvanceDisplayWindow, sets the token offset to its current value before moving the displayed window. Both sites use it.
• A unit test covers the unset and already-set cases. The token, mid-turn and offset tests pass, and lint is clean.
• 4 files, about +25/−2.

Next: mise run check is running (about 6–11 minutes). Once it passes I'll commit, push, resolve both findings with a note, and check the trail again. The trail was still reviewing, so more findings may arrive. CI on the first push is still running too.

Now waiting for CI and the trail's re-review on the new head: