question: so now we fixed if a branch is 100% included in the default branch (or another branch) but what if a branch is branched of the 10th commit (from HEAD) of main, how would the code handle this

Short answer: technically yes, but not in the streaming-relay sense git-sync is built around — it becomes a full materialize-translate-repack path, and it breaks signatures.

Why it isn't cheap:

1. Object contents embed hashes. Trees list entry SHA-1s, commits list parent/tree SHA-1s, tags list target SHA-1s. To produce SHA-256 objects you must decode every object, rewrite internal references, and rehash. That kills the relay optimization in bootstrap/sync — you can't just forward the source packfile bytes (internal/relay is the current fast path; see docs/bootstrap.md).

2. You need a persistent bidirectional mapping. For incremental runs, target tips are SHA-256 but the source's have negotiation expects SHA-1. git-sync would have to carry (or re-derive) a SHA-1↔SHA-256 map across runs, or re-translate everything each time. Git core handles this via extensions.compatObjectFormat / the "transition plan" loose-object mapping — doing it outside git means reimplementing that table.

3. Signed commits and signed tags break. The signature covers the object bytes. Rewriting references changes the bytes, so every GPG/SSH signature becomes invalid. There's no fix-up; this is inherent to content-addressed storage.

4. Memory/cost profile changes. Current --materialized-max-objects guardrail already bounds the fallback in-memory path. A translating push is strictly a materialized path plus a rewrite pass and a mapping table, so it's more expensive than today's worst case, not less.

5. Receive-pack on a SHA-256 target speaks SHA-256 wire format. That's a protocol switch too (the push path is still v1-style here; wire-level object IDs, ref advertisement, capabilities like object-format=sha256 all change). So the push side needs a second codepath, not a flag flip.

Practical take: it would be a separate --translate-to sha256 mode that replaces relay with decode-rewrite-rehash-repack, persists a hash map alongside managed refs, and documents that signatures don't survive. Worth it if the goal is a one-shot migration to a SHA-256 backend; not worth grafting onto the incremental relay path.

If you want, I can sketch where this would slot into the engine (plan → materialized translate stage → new push codec) without writing code yet.

Yes — this is actually a thing in git core, called the hash function transition / interop mode, using extensions.compatObjectFormat.

How it works server-side:

1. The target repo is created with objectFormat = sha256 and compatObjectFormat = sha1 (or vice versa). Git then stores a bidirectional hash map for every object it receives.
2. On receive-pack, if the client pushes SHA-1 wire format (negotiated via the object-format capability), the server translates incoming objects into its primary format and records the mapping. Trees/commits are rewritten so internal references point at the primary-format hashes; the original form is recoverable through the map.
3. On upload-pack/ref advertisement, the server can serve either format to a client depending on which object-format= the client negotiates. SHA-1 clients see SHA-1 hashes; SHA-256 clients see SHA-256.
4. Signed commits/tags: the transition plan handles this by signing in a canonical form and carrying the signature + original bytes; interop is designed so signatures still verify under the hash they were made in. (In practice this is the most brittle part and the bit operators hit issues with.)

The upshot for git-sync:

• If the target server actually implements compat mode, git-sync needs no translation code at all. You just push SHA-1 as today; the server does the work. git-sync's relay path even stays intact, because the wire bytes can flow through unchanged until they hit the server.
• You do need to negotiate object-format=sha1 in the push capabilities, and confirm the server advertises object-format at all. Today git-sync's receive-pack path doesn't inspect that; it would want to, just to fail loudly if the target is SHA-256-only with no compat.
• For incremental reruns, target ref advertisement comes back in whatever format the client asked for, so the have list stays SHA-1 and negotiation keeps working.

Current reality check (worth verifying against today's forges before relying on it):

• Git core has had this since ~2.42, marked experimental.
• GitLab has had some SHA-256 repo support; GitHub historically has not shipped SHA-256 repos.
• Gitea/Forgejo have varying support.
• Self-hosted git daemons are the most likely place you'd actually encounter a SHA-256 target with compat mode.

So: "translate on receive" is the right architectural layer for this problem, and it already exists in git — it's just a server capability question, not a git-sync feature request. The sensible thing for git-sync is to detect and surface it (probe object-format capabilities; refuse a SHA-256-only push target with a clear error) rather than try to implement translation itself.

Yes, later incremental sync works, as long as two conditions hold on the target:

1. Compat mode is permanent, not bootstrap-only. The SHA-1↔SHA-256 mapping has to stay enabled for the life of the repo, not just the initial import. Git does persist the map alongside objects, so this is the default once compatObjectFormat is set — but if an operator ever disables it later, the SHA-1 view disappears and your next sync has nothing to negotiate against.

2. git-sync always negotiates object-format=sha1 with the target. Both on the info/refs?service=git-receive-pack advertisement and on the push itself. That's what makes the server return ref tips as SHA-1 hashes (translated through the map) instead of its native SHA-256 form. Without this, target advertisement comes back as SHA-256 and is useless as have input to a SHA-1 source.

Given those, the incremental flow stays identical to today:

• Probe target → get SHA-1 tips (via compat map).
• Advertise those as have to source upload-pack.
• Source returns a thin SHA-1 pack of only new objects.
• Relay to target; server translates and extends the map.
• Fast-forward checks, --prune, --force, tag immutability — all operate on SHA-1 hashes on both sides, so the existing logic is untouched.

Edge cases that still behave correctly:

• Third-party pushes to the target in SHA-256 between syncs. The server computes compat SHA-1s for those objects too, so advertisement stays consistent. git-sync will see a SHA-1 tip it doesn't recognize; source won't have it as a have match, and the next fetch just won't benefit from it. No corruption, just a slightly less efficient negotiation.
• Ref rewrites on target. Hash comparison still works because both sides of the comparison are SHA-1.
• Tag retargeting. Same story — --force semantics are hash-based, format-agnostic.

One thing that does not work, and is worth flagging loudly: if compat mode was never enabled and the target is SHA-256-only, the first sync fails at capability negotiation, and no amount of later retrying fixes it without reconfiguring the target repo. So a sensible git-sync behavior would be to detect object-format=sha256 without object-format=sha1 in the receive capabilities on the very first probe and error out with a clear "target is SHA-256 only, enable compatObjectFormat=sha1 or this tool can't sync from a SHA-1 source" message — rather than discovering it mid-push.

Yes, with one nuance: it's two invocations, not one config. The tool is one-directional per run by design — source → target with explicit scope — so you orchestrate the split externally:

The scope flags already cover what you need per run:

• --branch filters which refs that invocation touches (cmd/git-sync/main.go:73).
• --map src:dst lets you rename during relay if the two sides use different conventions.
• --prune is per-invocation, so each direction only prunes refs it manages — the two runs don't step on each other as long as their --branch sets are disjoint.
• Fast-forward safety, --force, replicate mode, all behave the same regardless of which side you call "source."

The sharp edges to know about:

1. No wildcard branches. README is explicit: "Ref mapping is explicit, not wildcard-based." feature/* means enumerating branches, or listing them from a probe and piping into --branch. If bidirectional mixed-ownership becomes a common workflow, a --branch-pattern 'feature/*' flag on top of the existing planner filter is the obvious addition — small change, fits the existing scope model.

2. Ownership discipline is on you. Nothing in the tool prevents you from running both directions over overlapping branch sets and getting a ping-pong. The guardrail is your --branch list on each invocation. A lightweight config file that declares "these refs, this direction" and dispatches two runs would be a thin wrapper, not a core change — probably belongs in the caller (cron, CI, a small driver script), not in git-sync itself, to keep the CLI's semantics narrow.

3. replicate is directional too. If main should be strictly source-authoritative (overwrite target even on divergence), use replicate for that run; keep the feature-branch run as plain sync with fast-forward safety so target→source pushes from devs don't silently overwrite anything on the source side.

4. Auth is per-run. Both directions need credentials with push rights on their respective targets; GITSYNC_SOURCE_TOKEN / GITSYNC_TARGET_TOKEN get swapped between the two invocations. Trivial, but worth calling out if you're wiring this into CI secrets.

So: supported today for explicit branch lists; wants a glob flag if you want to express "all feature/*" declaratively; fundamentally the right shape because the split-ownership policy lives in the operator's config, not inside the sync engine.

Yes — and there's actually a bigger win hiding behind that suggestion. Ordering alone helps, but the real payoff is using the trunk's reachable set as a cutoff for every other branch's graph walk.

What the log is showing. planCheckpointsFromChain at internal/strategy/bootstrap/bootstrap.go:567 is called per branch in a plain loop (bootstrap.go:491). Each branch independently:

1. Fetches the full commit graph back to root (FetchCommitGraph, tree:0, no cutoff).
2. Walks first-parent to get a chain.
3. Plans checkpoints on that chain.

Nothing shares work across branches. For 152 feature branches that almost all descend from main, you're refetching and rewalking ~99% of main's history 152 times.

Tier 1 — cheap: default-branch-first ordering.

• Read HEAD symref from the source advertisement. Protocol v2 ls-refs returns it with the symrefs argument; v1 info/refs includes it in capabilities. git-sync already talks to the source at probe time, so no extra round-trip.
• Sort planRefs in bootstrap.go:465 so HEAD's branch is first; everything else stays in current order.
• Benefit at push time: the target's object store has trunk objects when feature-branch batches start, which means receive-pack handles subsequent batches as thin packs against a larger reachable set. This is free and small.

Tier 2 — the real win: trunk-reachability cutoff.

After planning trunk, you know every commit hash reachable from trunk's tip (you already compute the first-parent chain at bootstrap.go:581, and FetchCommitGraph materialized the full ancestor set). For every subsequent branch:

• Pass trunk tip as a have to FetchCommitGraph. Protocol v2 fetch honors haves; the source only sends commits not reachable from trunk. A "main + 5 commits" feature branch returns 5 commits instead of 75k.
• Short-circuit FirstParentChain when it hits a commit already in trunk's reachable set — no need to walk past the fork point. chain_len becomes divergence length, not absolute length.

Rough impact on your log output: today each branch takes ~150–400ms and chain_len is 40–128. Post-optimization, trunk is one ~normal planning pass, and each subsequent branch is probably ~20–50ms with chain_len in the single digits for the typical "branched off main last week" case. For 152 branches that's the difference between ~40s and ~5s on the planning phase alone, and the actual relay gets proportionally cheaper too because non-trunk batches shrink to their divergence deltas.

Things to handle cleanly.

• No clear trunk. If HEAD isn't advertised, or --branch/--map excludes it, fall back to current behavior. Alternatively pick the pivot as "branch with the longest graph" by doing one probe-time graph walk; probably not worth it — HEAD is the right heuristic 99% of the time.
• Branches with disjoint roots. The have-limited fetch just returns their full history; degrades gracefully.
• Shallow planner (bootstrap.go:602) already does incremental depth doubling per-branch; the same cross-branch reuse would apply there too — remember trunk's ancestry from the first full walk and stop shallow-depth doubling on other branches once they land inside it.
• Cache lifecycle. Trunk's reachable-set is the big thing to keep across branches. The current code explicitly nils graphStore and runtime.GC()s after extracting the chain (bootstrap.go:582) specifically because the full store is huge (4.6 GB for linux per the comment). You don't want to keep the full store around for 152 branches. But you only need the hash set of trunk's ancestry for cutoff purposes — that's 8-ish bytes per commit × ~75k commits = ~600 KB. Extract that set before you drop the store, carry it forward, drop it after planning finishes.

Where in the code it goes. The minimal version is: lift ordering into planBatches (bootstrap.go:489), compute trunk's reachability set on the first iteration, thread it as a cutoff into subsequent FetchCommitGraph calls and into FirstParentChain. That's two signature changes (FetchCommitGraph gains optional haves; FirstParentChain gains an optional "stop-at" set) and some wiring in the loop. No new concepts, just sharing what's already computed.

Worth a design note in docs/bootstrap-batching.md before implementing, since it changes the per-branch invariants the current code relies on. Want me to sketch that note?

Good — only bootstrap uses commit-graph fetches, so the single plumbing change is enough.

Quick status:

Implemented both tiers:

• Tier 1 — default-branch-first: orderTrunkFirst in bootstrap.go uses the new SourceHeadTarget param (derived from the v2 symrefs response or v1 symref= capability) to sort the trunk first; the rest of the order is preserved.
• Tier 2 — trunk-reachability cutoff: planCheckpointsFromChain now takes trunkHaves []plumbing.Hash (passed as haves to FetchCommitGraph, so the source skips commits reachable from trunk) and trunkStopAt map[plumbing.Hash]struct{} (passed to the new FirstParentChainStoppingAt, so the walk terminates at trunk ancestors). The trunk's full commit set is extracted from the graph store via collectCommitHashes before the store is GC'd — ~8 B × commit count, so ~11 MB for linux vs the ~4.6 GB store.

Files touched:

• internal/gitproto/refs.go — HEAD symref extraction (v1 via capability.SymRef, v2 via symrefs ls-refs arg); new RefService.HeadTarget field.
• internal/gitproto/fetch.go — FetchCommitGraph takes a haves slice.
• internal/planner/checkpoint.go — added FirstParentChainStoppingAt, kept FirstParentChain as a thin wrapper.
• internal/strategy/bootstrap/bootstrap.go — new SourceHeadTarget param, orderTrunkFirst, trunk cutoff wiring, empty-chain fallback for subsumed branches.
• internal/syncer/syncer.go — plumbs s.sourceService.HeadTarget into bstrap.Params.
• Tests: internal/planner/planner_test.go, internal/gitproto/refs_test.go, internal/gitproto/fetch_test.go, internal/strategy/bootstrap/bootstrap_test.go — new unit tests for stop-set walk, orderTrunkFirst ordering edge cases (HEAD filtered out, no HEAD hint, non-zero trunk index), v1 HEAD symref parsing, and v2 ls-refs symref-target attribute parsing.

Behavior when HEAD is unknown or filtered out by --branch: falls back to existing per-branch behavior; no regression.

go build ./... and go test ./... both green. The gated e2e integration tests (GITSYNC_E2E_GIT_HTTP_BACKEND=1, GITSYNC_E2E_LIVE_LINUX=1) will exercise the new path end-to-end when run.

Want me to add a note to docs/bootstrap-batching.md describing this change before committing?

All green. Summary:

Fix 1 — the silent bug. listSourceRefsV2 now always adds ref-prefix HEAD to the ls-refs args, and decodeV2LSRefs extracts the symref target from HEAD and then drops HEAD from the returned ref slice (matches v1 behavior via the symref capability). Previously, because planner.RefPrefixes always passes refs/heads/, HEAD was filtered out of the v2 response entirely and HeadTarget was always empty on v2 sources — exactly what your log showed.

Fix 2 — observability. planBatches now emits one of three log lines up front:

• bootstrap batch trunk unset reason="source did not advertise a HEAD symref" — source gave us nothing (rare but possible on minimal git servers).
• bootstrap batch trunk unset reason="HEAD ref not in desired set (filtered by --branch or --map)" source_head_target=refs/heads/main — source advertised HEAD but the user's scope excluded it.
• bootstrap batch trunk selected source_head_target=refs/heads/main trunk_target_ref=refs/heads/main — fast path active; the first branch in the checkpoint logs after this should show have_count=1 stop_at_count=<trunk_ancestry>.

Re-run your command and you should now see the trunk-selected line, then the first branch planning with have_count=0 stop_at_count=0 (trunk itself), then subsequent branches with have_count=1 stop_at_count=<big number> and drastically shorter chain_len values.

Short answer: yes, expect roughly the same webhook volume — the optimization I implemented is entirely on the planning side, not the push side, and GitHub fires one webhook event per ref update regardless of how those pushes are batched.

Here's the precise picture:

What the trunk-first change reduces

• Source-side commit-graph fetches (smaller, faster, fewer objects).
• FirstParentChain walk length per branch.
• Objects actually transferred in later batch packs (because have-negotiation via completed branch tips, already in place since 3c20da5).

What it does not reduce

• Number of receive-pack invocations against the target.
• Number of ref updates.
• Number of webhook deliveries from GitHub.

The batched bootstrap's push pattern is still: for each branch, push its checkpoint pack(s) to refs/gitsync/bootstrap/heads/<branch>, then cut over to the real refs/heads/<branch>. With 152 branches that's still ~152 temp-ref updates plus ~152 cutover updates, and GitHub generates a push event per ref-update, not per HTTP call.

Three levers you could actually pull to cut webhook volume

1. Disable the webhook on the target during bootstrap. By far the easiest operationally: most teams do exactly this for migrations and large one-shot imports. Re-enable after the final cutover batch. No code changes needed.

2. Skip the pack push for fully-subsumed branches. My trunk-first change already produces chain=[tip] for branches whose tip lives inside trunk's ancestry. Today that still does a tiny fetch + push. We could detect this case (branch tip is in the trunk reachable set) and emit a single ref-creation command with no pack at all — that's still a webhook, but fewer HTTP calls and no spurious "0 objects" pushes. Small win.

3. Batch cutover ref updates into a single receive-pack call. receive-pack accepts N ref-update commands in one push. Doing one HTTP call with 152 cutover commands instead of 152 separate calls would cut HTTP overhead a lot — but GitHub still fires 152 push events, one per ref. So this helps throughput, not webhook noise.

If webhooks are the real pain point, option 1 is the right answer; the tooling can't do better than git's per-ref event model. If you want me to implement option 2 (skip-pack for subsumed branches) or add a --disable-target-webhooks helper that uses the GitHub API to toggle them around the run, say the word — both are reasonable follow-ups but not part of the current planning optimization.

No — we already have exactly what we need, and it's the same ~11 MB hash set the trunk-first optimization already keeps in memory.

Look at the current flow in bootstrap.go:

1. Fetch trunk's commit graph into graphStore (~4.6 GB transient for linux).
2. collectCommitHashes(graphStore) — extracts just the hashes into ancestors (~8 B × commit count ≈ 11 MB for linux).
3. Drop graphStore, GC reclaims the 4.6 GB.
4. ancestors becomes trunkStopSet, carried across the per-branch loop.

trunkStopSet already contains every commit reachable from the trunk tip (not just the first-parent chain — collectCommitHashes iterates all CommitObjects in the store, and the tree:0 fetch returns the full reachability). So "is this branch tip fully subsumed by trunk?" is a one-line O(1) check against a set we already have:

That's it. No additional memory. In fact it makes things cheaper than the current code path, because today when a branch is subsumed my code still fetches its commit graph (which returns near-nothing via the have-negotiation but still round-trips) and does a tiny push. Option 2 would skip both.

The only genuine caveat — not a memory one — is correctness edge cases:

• Branch tip == trunk tip: trivially in the set, skip is safe. Webhook fires for the ref creation, but no pack overhead.
• Branch is an older commit on trunk's first-parent line: in the set, skip is safe.
• Branch is reachable only via a non-first-parent merge on trunk: still in the set (because collectCommitHashes pulls every commit the tree:0 fetch returned, including second parents and their ancestors), so skip is still safe.
• Branch tip not reachable from trunk at all: not in the set, falls through to the current planning path. Correct.

So the question is really about code surface, not memory. Want me to implement it as a follow-up to the current change?

Done. Summary of what option 2 adds:

Planning-side (planBatches): before each non-trunk branch's commit-graph fetch, check trunkStopSet[ref.SourceHash]. On hit, emit a plannedBatch{subsumed: true} with just a BranchPlan (no TempRef, no Checkpoints, no chain). Log line: bootstrap batch branch subsumed by trunk branch=<ref> source_hash=<sha>.

Execution-side (executeBatched): at the top of the batch loop, branch on batch.subsumed. On hit: single PushCommands call with one ref-create at SourceHash, update completedRefs, increment BatchCount, log bootstrap batch subsumed branch finalized. No temp ref, no pack fetch, no pack push, no cutover-then-delete dance.

Cost saved per subsumed branch: one commit-graph fetch + one pack fetch + one pack push + one temp-ref-update + one cutover push + one temp-ref delete → replaced by a single ref-create command. On webhooks you still get one event for the ref creation; everything else is pure network/CPU savings.

Correctness: trunkStopSet is built from every CommitObject in trunk's fetched graph (not just first-parent), so any branch tip that's reachable through trunk's merge parents is also detected as subsumed. Non-subsumed branches flow through the unchanged original path. If there's no trunk (HEAD unavailable or filtered out), trunkStopSet is nil and the subsumed check is skipped entirely — no regression.

Test TestExecuteBatchedSubsumedBranchSkipsPack in bootstrap_test.go verifies: fetchCommitGraph is called once (trunk only), fetchPack is called once (trunk only), PushPack is called once (trunk only), and the feature branch's ref is created by a single PushCommands with Old=0, New=featureHash, Delete=false.

All tests green.

All green. Two meaningful commits on top of b0f3048:

1. c95cea3 — Plan trunk first in batched bootstrap to cut per-branch graph fetches

   • HEAD symref discovery (v1 via capability.SymRef, v2 via symrefs + the ref-prefix HEAD fix)
   • FetchCommitGraph haves param
   • FirstParentChainStoppingAt walk primitive
   • orderTrunkFirst, collectCommitHashes, SourceHeadTarget plumbing, trunk-selected/unset diagnostic logs
   • 389 insertions / 48 deletions across 9 files

2. 411c469 — Skip pack push for branches fully subsumed by trunk

   • subsumed flag on plannedBatch
   • Planning-side detection before the commit-graph fetch
   • Execution-side single-ref-create handling
   • Webhook-volume caveat in the commit message since that's the operational question you asked
   • 126 insertions across 2 files

Note: a pre-commit hook moved HEAD onto a new branch soph/branch-optimizations on the first commit. Your starting branch soph/plan-optimization is untouched. If you want these on the original branch instead, say the word and I'll reset and re-apply.