Compact Transcript Storage and Boundary Markers

Claude Code·Opus 4.8·

Soph·2mo ago·6hr 13min·10 Checkpoints·20 file changes·+655/-160·913.7K tokens

Here's the summary, based on tracing the condensation and storage code.

Short answer

Neither format stores only deltas in the git sense — but they differ in scope. Both are written as complete files per checkpoint; the difference is how much of the session each one contains:

full.jsonl = the entire session transcript, from line 0 to the end. The checkpoint boundary isn't baked into the file — it's recorded separately as a line offset (checkpoint_transcript_start) in that session's metadata.json.
transcript.jsonl (the new compact format) = the transcript pre-sliced to checkpoint_transcript_start → EOF, then run through transcript/compact. So it physically contains only the slice belonging to that checkpoint cycle — effectively a per-checkpoint delta, already trimmed and normalized.

So the answer to your question: the full log carries the whole thing + a start position; the compact log carries only the slice (delta) and doesn't need a start marker because the slicing already happened when it was written.

How it's laid out

Each committed checkpoint gets its own directory on entire/checkpoints/v1, with one sub-dir per session:

metadata.json sessions[] carries two pointers:

transcript → always full.jsonl (CLI rewind/resume/explain resolve by this filename)
compact_transcript → transcript.jsonl, omitted when the compact one wasn't generated

The pivot: `checkpoint_transcript_start`

It's a 0-indexed line offset (not bytes, not message index), tracked in SessionState:

starts at 0, and after each condensation is bumped to the transcript's current line count ("everything since the last checkpoint").
When writing the compact file, transcript.SliceFromLine(fullBytes, CheckpointTranscriptStart) chops off everything before it, then Compact() normalizes the remainder.

So e.g. if a checkpoint's start = 750 on a 1000-line transcript: full.jsonl stores lines 0–999 (with start=750 in metadata), while transcript.jsonl stores a compacted lines-750–999 only.

Two things worth knowing

transcript.jsonl is best-effort. It's skipped if compaction fails, produces empty output, or exceeds the max chunk size — so a checkpoint may have full.jsonl but no compact_transcript. full.jsonl is the source of truth.
Per-checkpoint copies, deduped by git. Each checkpoint dir gets its own full copy (intentionally self-contained), but full.jsonl is content-hashed (content_hash.txt = sha256) and git's blob store collapses identical ranges, so unchanged history isn't physically duplicated on disk despite appearing in every checkpoint's tree.

Net: the system is full-log + offset as the canonical store, with the compact format being a pre-sliced, pre-compacted convenience view of just that checkpoint's slice.

Confirmed. Yes — you'd need a second marker, and the important part is it can't be the same number you already have.

Why

checkpoint_transcript_start is a line offset into the raw full.jsonl, and today the slice happens before compaction (compact.go:106, SliceFromLine(content, opts.StartLine) then compact). If you instead stored the full compacted transcript, you'd need a marker pointing at the checkpoint boundary inside the compact file — and that's a different coordinate space, because compaction is not line-preserving. From this same code:

streaming assistant fragments with the same message ID get merged into one line (N raw → 1 compact),
tool_result lines get inlined into the preceding assistant's tool_use block (a line disappears),
whole types are dropped (droppedTypes: progress, file-history-snapshot, queue-operation, system).

So raw line 750 might be compact line ~300, with no fixed ratio. The existing offset is useless for indexing into the compact file.

And it's worse than "just a second int"

For several agents StartLine isn't even a line offset in the first place — it's a message/item index interpreted per-format:

OpenCode / Gemini (single-object formats): message-index offset
Codex: response_item index
pi: handled internally by compactPi

So a "full compact transcript + boundary marker" design would need that marker computed in compact-output coordinates per format, produced by the compaction pass itself (e.g. emit the boundary line index while writing, or count compacted lines for the pre-start portion). You can't derive it after the fact from checkpoint_transcript_start.

The tradeoff

That's essentially why the current design slices-then-compacts instead: storing the delta sidesteps the coordinate-mapping problem entirely — no second marker needed, and each checkpoint's compact file is self-contained. Going to full-compact buys you a single deduplicated compact transcript per session, but costs you (a) a new per-format boundary marker emitted during compaction, and (b) all read paths learning to seek to it. If the goal is dedup, note git already collapses identical blob ranges across checkpoints, so the on-disk win may be smaller than it looks.

Verified the crux: in manual_commit_condensation.go:220,270, the Transcript written to each checkpoint is sessionData.Transcript — the whole transcript to date, not a slice. Only the compact transcript.jsonl gets sliced via StartLine. That changes your premise a bit, so let me separate what's actually true today from what your argument is really about.

Today, the two formats have different loss profiles

full.jsonl is already full-per-checkpoint. Every checkpoint re-stores the entire transcript-to-date (redacted). So losing a middle checkpoint loses nothing raw — the latest surviving checkpoint is a superset. The only raw data you can lose is the tail: if the most recent checkpoint(s) get dropped, you fall back to the newest survivor and lose everything after it.
transcript.jsonl (compact) is the fragile one. It's per-checkpoint slices, so reconstructing the full compact view means concatenating all slices in order. Lose a middle one → a gap. (Though you can always regenerate it from full.jsonl, since that's intact.)

So your recoverability worry is real, but it's specifically a property of the compact/delta format, not the session as a whole. The raw session survives a mid-history loss today.

On rebase specifically

Worth being precise: checkpoints live on the separate permanent entire/checkpoints/v1 branch, not on your working branch. A working-branch rebase doesn't delete v1 commits — what breaks is the linkage (which commit ↔ which checkpoint), which the post-rewrite hook tries to realign. So "the whole session becomes unrecoverable on rebase" overstates it for the current design; the content is still on v1, it's the attachment that's at risk. Revert is similar — the v1 data stays.

Your real proposal, and why it's sound

What you're describing — make transcript.jsonl full-per-checkpoint (with a boundary marker per checkpoint instead of a slice) — is just making the compact format adopt full.jsonl's existing model. That's consistent with your own design constraint from the carry-forward work: "duplicate data on the checkpoints branch is acceptable." Benefits line up exactly with what you said:

Every checkpoint self-contained → robust to any middle loss, no piecing-together, regeneration unnecessary.
UI read model → your "read the last checkpoint + just positions from the others" works if the last is identifiable and survived. But the strictly-more-robust version of that same idea is "read any surviving checkpoint for content, use the markers to segment" — which is what self-containment buys you, and it doesn't depend on the tail being intact.

The two costs to weigh

The marker must be in compact coordinates (the per-format problem from before): you can't reuse checkpoint_transcript_start (raw line offset) — compaction merges/inlines/drops lines, and several agents treat StartLine as a message/item index. So the boundary has to be emitted during compaction, per format. That's the real implementation cost of "full + position marker."
Storage isn't O(n) — but packing saves you. Full-per-checkpoint means a growing prefix stored N times → O(n²) in loose objects. The earlier "git dedups identical ranges" framing I gave is wrong: git dedups whole identical objects, and a growing transcript is a different object each time. What actually rescues it is packfile delta compression (git gc) — checkpoint N+1's full.jsonl packs as a small delta against N's. So the cost is real for loose objects and largely reclaimed once packed. You already pay this for full.jsonl; mirroring it on the (smaller) compact format is the same shape, smaller constant.

Net: your instinct is right for the compact format — full-per-checkpoint + a compact-space boundary marker is more robust and gives the cleaner UI read than delta slices. The thing to budget for is computing that marker inside each format's compaction path, not the storage.

Done. The work is implemented, tested, lint-clean, and committed in three logical layers on a new branch.

Summary

Branch: soph/full-compact-transcript-per-checkpoint (off main)

What changed: transcript.jsonl now stores the full compacted session on every checkpoint (mirroring how full.jsonl already works), with a compact_transcript_start marker in session metadata recording where each checkpoint's slice begins. This makes every checkpoint self-contained — a session is reconstructable from any single surviving checkpoint, robust to a mid-history checkpoint being lost, reverted, or dropped in a rebase — and lets the UI read one file and segment by the marker instead of stitching deltas.

The three commits (each builds on the last):

compact: add FullWithBoundary — compacts the whole transcript + returns the boundary as lineCount(full) − lineCount(delta), reusing every format's existing slicing logic (no rewrite of the 7 emitters). 4 new tests, incl. an index-format (Gemini) case.
api/checkpoint: add CompactTranscriptStart marker — *int so nil (legacy delta file) is distinguishable from 0 (full file, first checkpoint), plus GetCompactTranscriptStart(). Round-trip + getter tests.
checkpoint: store full compact transcript per checkpoint — wires the boundary through both the normal write and the pre-push OPF rewrite path; updates the two old delta-semantics tests to assert the new full-content + marker contract.

Verification: mise run fmt + mise run lint clean (0 issues); checkpoint, compact, api/checkpoint, fsstore, and strategy test suites all pass.

Known tradeoff (documented in code): a deterministic off-by-one when a single logical message (a streaming assistant message, or a tool_result carrying text) straddles the exact StartLine boundary — harmless for segmentation.

Two notes for when you pick this up:

In-repo readers only write/point at transcript.jsonl; the actual consumer that segments by the marker is downstream (entire.io), which will need to read compact_transcript_start and slice — and treat its absence as "legacy, read from line 0."
I haven't pushed or run the full test:e2e:canary — say the word and I'll run the canary and/or push.

/simplify → 4 cleanup agents in parallel → apply the fixes

You are improving the quality of the changed code, not hunting for bugs. Review it for reuse, simplification, efficiency, and altitude issues, then fix what you find. Do not look for correctness bugs — that is what /code-review is for.

Phase 0 — Gather the diff

Run git diff @{upstream}...HEAD (or git diff main...HEAD / git diff HEAD~1 if there's no upstream) to get the unified diff under review. If there are uncommitted changes, or the range diff is empty, also run git diff HEAD and include the working-tree changes in scope — the review often runs before the commit. If a PR number, branch name, or file path was passed as an argument, review that target instead. Treat this diff as the review scope.

Phase 1 — Review (4 cleanup agents in parallel)

Launch 4 independent review agents via the Agent tool, all in a single message so they run concurrently. Pass each agent the diff and one of the four angles below. Each returns its findings with file, line, a one-line summary, and the concrete cost (what is duplicated, wasted, or harder to maintain).

Reuse

Flag new code that re-implements something the codebase already has — Grep shared/utility modules and files adjacent to the change, and name the existing helper to call instead.

Simplification

Flag unnecessary complexity the diff adds: redundant or derivable state, copy-paste with slight variation, deep nesting, dead code left behind. Name the simpler form that does the same job.

Efficiency

Flag wasted work the diff introduces: redundant computation or repeated I/O, independent operations run sequentially, blocking work added to startup or hot paths. Also flag long-lived objects built from closures or captured environments — they keep the entire enclosing scope alive for the object's lifetime (a memory leak when that scope holds large values); prefer a class/struct that copies only the fields it needs. Name the cheaper alternative.

Altitude

Check that each change is implemented at the right depth, not as a fragile bandaid. Special cases layered on shared infrastructure are a sign the fix isn't deep enough — prefer generalizing the underlying mechanism over adding special cases.

Phase 2 — Apply the fixes

Wait for all four agents to complete, dedup findings that point at the same line or mechanism, and fix each remaining one directly. Skip any finding whose fix would change intended behavior, require changes well outside the reviewed diff, or that you judge to be a false positive — note the skip rather than arguing with it. Finish with a brief summary of what was fixed and what was skipped (or confirm the code was already clean).

2mo ago·2m

/simplify complete.

Applied (2 fixes, committed as `4a360d842`)

Reuse — Extracted updateSessionMetadata, the read‑modify‑write‑blob pattern that setCompactTranscriptStart (new) and replaceSkillEvents (existing) both duplicated (~22 lines). Both now express only their field mutation via a lambda. Kept the read‑from‑blob semantics so sequential finalize‑path mutations still see each other's changes. (This also subsumes the efficiency agent's OPF re‑read finding — deliberately not optimizing the read away, since the blob read is what keeps it correct.)
Simplification — Generalized the test helper assertCodexSliceIsGamma → assertCompactSliceScoped(t, content, marker, wantAbsent, wantPresent), now shared by both the Claude and Codex slice‑scoping assertions (removed an inline copy in the third test).

Skipped (with reasons)

boundary < 0 clamp — cheap defensive guard on an intentionally approximate value (the documented straddle off‑by‑one); removing it risks a negative slice index downstream.
StartLine <= 0 early return — it's a real efficiency win, skipping the second full compaction on the very common first‑checkpoint case; removing it would regress.
Drop the writeTranscript bool — false positive: the non‑compactable path returns wrote=true, compactStart=nil, so the bool is not derivable from the pointer.
Double compaction in FullWithBoundary — all agents agreed this is the acknowledged tradeoff vs. rewriting 7 per‑format emitters; acceptable.
Cross‑package nonEmptyLines — the reuse agent itself recommended skipping (not worth the import coupling).

Altitude review: clean — no findings; the marker mechanism, *int discriminator, and dual write paths were all judged to be at the right depth.

Verification: go build ./... OK, mise run lint 0 issues, checkpoint/compact/api·checkpoint/fsstore test suites all pass. The branch now has 4 commits.

P2 cmd/entire/cli/transcript/compact/compact.go:158: FullWithBoundary derives the marker from compacted line counts. If StartLine lands between streaming assistant fragments with the same message ID, Compact merges the full transcript into one assistant line while the delta also has one line, so the marker becomes 0 and fullCompactLines[marker:] includes pre-checkpoint content. This violates the new compact_transcript_start slicing contract. A regression test with two same-ID assistant fragments split at StartLine=1 would catch it.
- P3 api/checkpoint/metadata.go:433 and docs/architecture/sessions-and-checkpoints.md:209: exported comments/docs still say transcript.jsonl is checkpoint- scoped and needs no offset. This branch changes it to full-session compact output plus compact_transcript_start, so downstream readers following the current docs will over-read old checkpoint content.

2mo ago·7m

Both review findings addressed and committed (110390a1b).

P2 — straddling-boundary rounding (real contract gap)

Confirmed with a probe: when StartLine falls between two same-ID streaming assistant fragments, compaction merges them into one line, boundary=0, and fullCompactLines[0:] includes the pre-checkpoint fragment.

The key realization: no integer line offset can split within a merged line, so exactness is impossible without the per-formatter rewrite we explicitly avoided. The rounding I have always includes the straddling line — it over-reads by ≤1 merged line but never drops this checkpoint's content (the alternative, rounding the other way, would lose FRAG_B). That's the least-bad behavior, so I:

Pinned it with a regression test (TestFullWithBoundary_StraddlingAssistantFragments_RoundsToInclusion) asserting boundary=0 and that the merged line is retained.
Rewrote the FullWithBoundary contract comment (dropped the misleading "harmless") and added a "tolerate bounded head overlap" note to CompactTranscriptStart so downstream segmenters handle it.

P3 — stale docs

Updated all four sites that still called transcript.jsonl "checkpoint-scoped / pre-sliced": the metadata.go field comment + directory-layout diagram, docs/architecture/sessions-and-checkpoints.md, and CLAUDE.md/AGENTS.md. They now state it stores the full compacted session, that consumers slice at compact_transcript_start (compact-output coordinates — distinct from the raw-line checkpoint_transcript_start), that a nil marker means a legacy delta-only file, and the bounded-overlap caveat.

Verified: build OK, lint 0 issues, the new straddle test + all compact/checkpoint/api·checkpoint suites pass. Branch is now 5 commits.

One thing worth flagging for the downstream (entire.io) consumer: it must read compact_transcript_start and slice, treat absent as legacy/read-from-0, and tolerate the ≤1-line head overlap — that's now documented but not yet implemented on their side.

Both fixed and committed (799bb0ad7):

paths.go:32 — CompactTranscriptFileName now says it holds the full compacted session, sliced at compact_transcript_start.
persistent.go:747 — writeTranscript doc updated: writes the full compacted session and returns the compactStart offset (no longer "not yet referenced by metadata").

I also swept the whole repo for pre-sliced / scoped to this checkpoint / not yet referenced and confirmed nothing else is stale. The one remaining scoped to this checkpoint hit (attribution.go:619) is about prompt.txt scoping — a separate, still-accurate concept — so I left it.

Verified: build OK, lint 0 issues, checkpoint + paths suites pass. Branch is now 6 commits.

No — no PR yet. The work is all committed locally on soph/full-compact-transcript-per-checkpoint (6 commits), but I haven't pushed the branch or opened a PR.

Want me to push and open one? A couple of things worth confirming first since they affect the PR:

Full mise run test:ci — I've run the affected packages (checkpoint, compact, api/checkpoint, strategy) but not the whole CI suite (unit + integration + Vogon canary). I'd run that before pushing per the repo's pre-push rule. (E2E with real agents I'd leave off unless you ask.)
Downstream caveat — the PR body should call out that the entire.io reader still needs to consume compact_transcript_start (and treat absent = legacy/read-from-0, tolerate the ≤1-line head overlap). That's a follow-up not in this PR.

Say the word and I'll run test:ci, push, and open the PR (with a summary covering the design, the straddle-rounding tradeoff, and the downstream follow-up). Want me to go ahead?

PR opened: https://github.com/entireio/cli/pull/1581

Summary of what I did:

mise run test:ci — first run hit a known intermittent flake (a transcript-unrelated package; the documented go-git reconcile flake); re-ran clean: 0 failures, Vogon canary roger-roger, integration + unit all green.
Pushed soph/full-compact-transcript-per-checkpoint (6 commits) to origin.
Opened PR #1581 against main with a body covering the what/why/how, the straddle-rounding tradeoff, the test coverage, and the downstream entire.io follow-up.

The PR is ready for review. The one open item it calls out is the downstream reader work in entire.io (consume compact_transcript_start, treat absent as legacy, tolerate the ≤1-line head overlap) — that's intentionally a separate change.

Done. Here's the full picture.

PR comments (Copilot — 3 inline, no human reviews yet)

OPF stale compact (persistent.go:1822) — fixed.
Double compaction (compact.go:163) — accepted tradeoff, explained.
Oversized-drop on long sessions (persistent.go:900) — documented as known limitation (your call).

Trail #707 — 1 open finding

Same oversized-drop issue as Copilot #1 — dismissed with rationale (pointing at the doc + the OPF fix). Trail is now 0 open findings.

What I changed (committed `05e15705a`, pushed)

#3 fix: replaceTranscript now drops the prior transcript.jsonl and clears CompactTranscriptStart when compact regeneration produces nothing — so the OPF rewrite can't leave a stale, less-redacted compact or a marker pointing at mismatched content. setCompactTranscriptStart takes *int to allow clearing; added regression test TestUpdateCommitted_DropsStaleCompactWhenRegenerationProducesNone. (Existing finalize/Codex tests still pass — success path intact.)
#1 doc: documented the 50MB-cap limitation in sessions-and-checkpoints.md and CLAUDE.md, and corrected the now-stale "finalization keeps the previous transcript.jsonl" clause (it now drops it).
Posted a consolidated reply on the PR addressing all three Copilot points.

Verified: build OK, lint 0 issues, checkpoint/compact/api·checkpoint suites green. Branch is now 7 commits.

Net: the only substantive code fix the reviewers surfaced (#3, privacy-relevant) is done; #1 is a deliberate, documented known-limitation with a clear follow-up (chunk the compact like full.jsonl) if it ever matters; #2 was already an informed tradeoff. Nothing left open on the trail.