can you look at the latest changes for the replicate command, can you look at https://git-scm.com/docs/bitmap-format and I wonder if this could help with some things?

I’m reviewing the recent replicate command changes locally, then I’ll read the Git bitmap format docs and check whether any of that machinery plausibly helps this implementation.

I’ve identified the replicate-related merge at b0f3048 and the main implementation/test files. Next I’m diffing that stack to separate behavior changes from plumbing, then I’ll map those against what Git bitmaps actually accelerate.

The commit range inspection hit a promisor-object fetch, so the local clone doesn’t have the full objects for that diff. I’m rerunning those reads with network access so I can review the actual patch rather than guessing from filenames.

The core shape is clear now: replicate is pure relay, uses source upload-pack plus target receive-pack, disallows --force, and tolerates no-thin by always fetching a self-contained pack. I’m drilling into the adaptive batching path next, because that’s where bitmap/graph acceleration would matter if it matters at all.

Findings

No concrete correctness bugs stood out in the current replicate path from the code I could inspect at HEAD.

Residual risk: this repo looks like a partial clone, and git diff 3a8d57c..b0f3048 wanted to fetch missing promisor objects from GitHub. I reviewed the current implementation and tests instead of the exact historical patch range.

What Changed

The latest replicate work is centered in internal/syncer/syncer.go, internal/strategy/replicate/replicate.go, internal/planner/relay.go, and internal/strategy/bootstrap/bootstrap.go.

At b0f3048 on April 14, 2026, replicate became a source-authoritative, relay-only mode that:

• allows overwrite-style branch/tag updates via relay,
• rejects --force and tells users to use sync instead,
• accepts no-thin targets because fetches are deliberately self-contained,
• falls through to bootstrap when the managed target refs are absent,
• carries TargetMaxPackBytes through that bootstrap path, including adaptive checkpoint subdivision and resume handling.

The important implementation point is that replicate itself is simple: it fetches a pack from source using current target refs as haves, pushes create/update refs via PushPack, then sends deletes separately via PushCommands replicate.go. Most of the heavy lifting is in the batched bootstrap planner, which currently fetches a commit-only graph (filter tree:0), walks the full first-parent chain in memory, then estimates/splits batches bootstrap.go.

Bitmap Format

Short answer: yes, but mostly indirectly.

Git’s bitmap format is an on-disk reachability index for a pack or MIDX, not a wire-protocol feature. The docs say bitmaps store reachability information for objects in a packfile or MIDX, and git rev-list --use-bitmap-index can use them to speed traversal when available:

• https://git-scm.com/docs/bitmap-format
• https://git-scm.com/docs/git-rev-list

My inference: bitmaps can help the source server generate fetch packs faster, so they may already improve replicate with zero code changes when the source is native Git and has fresh pack/MIDX bitmaps. But they do not directly solve this codebase’s main expensive bootstrap-planning step, because that step is client-side and needs a first-parent chain after downloading commit data bootstrap.go.

Where bitmaps could help:

• Faster source-side pack construction for replicate and per-checkpoint bootstrap fetches, if the remote Git server uses them.
• Potentially better local estimation, but only if you change architecture to operate on a local native-Git repo and call bitmap-aware commands like git rev-list --use-bitmap-index or git pack-objects.

Where they likely do not help much in the current design:

• Reading bitmap-format directly from this client over HTTP.
• Deriving the first-parent chain for checkpoint placement.
• Avoiding the transient in-memory commit-graph store described in bootstrap.go.

If you want, I can do a second pass specifically on “how to exploit native Git bitmaps here” and sketch the lowest-risk integration points.

Yes, but only in a narrow sense.

You can exploit native Git bitmaps without local disk only if you let the source-side native Git server use them for pack generation. In that model, git-sync does not read bitmap files itself; it just sends normal fetch requests, and upload-pack on the remote benefits from its own bitmap/MIDX indexes. Git’s bitmap docs describe them as pack/MIDX reachability indexes, and git rev-list documents bitmap-accelerated traversal via --use-bitmap-index:

• https://git-scm.com/docs/bitmap-format
• https://git-scm.com/docs/git-rev-list

What that means here:

• replicate incremental relay can benefit already, indirectly. Your FetchPack(... desired, targetRefs) call in internal/strategy/replicate/replicate.go is exactly the kind of fetch a native upload-pack may accelerate with bitmaps.
• Batched bootstrap fetches can also benefit indirectly, because each checkpoint fetch in internal/strategy/bootstrap/bootstrap.go is still just a normal fetch with want/have.

Where bitmaps do not help in your current no-disk client architecture:

• They do not give you a protocol-level way to ask the remote for “first-parent chain only”.
• They do not eliminate the expensive client-side planning step in internal/strategy/bootstrap/bootstrap.go, where you fetch commit objects into memory and walk the chain yourself.
• They are not exposed over smart HTTP as a reusable artifact your go client can consume directly.

So the realistic answer is:

1. Best no-disk win: rely on remote native Git bitmaps automatically.
2. What to change in git-sync:
   • Detect when the source is likely native Git and note that server-side bitmaps may help.
   • Benchmark fetch latency/object count for replicate and checkpoint fetches, but do not add bitmap-specific client logic.
3. What not to spend time on: parsing .bitmap format in this client. Without a local repo or a new custom server API, it won’t unlock the planning bottleneck.

If you want actual leverage on the bootstrap planner without local disk, the better direction is probably changing the checkpoint-planning algorithm, not chasing bitmap integration. For example:

• plan batches from observed pack/object counts during adaptive subdivision,
• cache prior branch-chain results across retries,
• or add a source-side helper endpoint if you control the server.

My inference from the docs and your code: native Git bitmaps can speed the remote fetches, but they probably cannot solve the expensive in-memory bootstrap planning step unless you introduce a local native-Git repo or a custom server capability.

It means: stop trying to predict batch boundaries up front from a fetched commit graph, and instead learn the right batch size from the packs the source actually produces.

Right now batched bootstrap does this in internal/strategy/bootstrap/bootstrap.go:

• fetch commit-only graph,
• build full first-parent chain,
• estimate batch count from chainLen * estimatedBytesPerCommit,
• place even checkpoints,
• then correct mistakes later with pack-header checks and target-rejection subdivision.

The alternative is more feedback-driven:

1. Start with a coarse guess.
2. Fetch one candidate checkpoint pack.
3. Read the PACK header object count, or measure actual streamed bytes if you can.
4. Use that observed density to predict the next checkpoint distance.
5. If the push is too large, bisect smaller.
6. If it is comfortably under limit, make the next chunk larger.

So instead of “64 KiB per commit for everyone”, you derive a repo-specific estimate like:

• this branch segment produced 420,000 objects for 9,000 commits
• or 310 MiB for 4,500 commits
• therefore next checkpoint should be around half / 1.3x / etc.

The point is that your current planner pays a huge cost to fetch the whole commit graph just to guess batch boundaries. An observed-count approach would trade that for iterative probing with real pack outputs.

The hard part is the phrase “during adaptive subdivision” only makes sense if you can still choose new checkpoints. For that you need some way to map “smaller slice” to an older commit. Today you have that because you already materialized the full first-parent chain. Without that chain, adaptive sizing alone is not enough.

So the fuller statement is:

• If you keep the chain, observed pack/object counts can replace the fixed estimatedBytesPerCommit heuristic and reduce bad initial splits.
• If you want to remove the chain fetch entirely, observed counts are not sufficient by themselves; you also need a cheaper way to discover candidate ancestor commits for checkpointing.

That is why I said it as a direction, not a drop-in fix. In this codebase, the pragmatic near-term version would be:

• keep the existing chain-based planner,
• replace the static 64 KiB/commit estimate with rolling observations from earlier batches on the same repo/branch,
• use those observations to choose better initial checkpoint spacing on later branches or retries.

I’m checking the actual Git-side options for finding a branch’s root or first-parent base, then I’ll map those back to what is and isn’t available over the protocol you’re using.

Maybe, but not via stock fetch protocol in a clean no-disk client.

If by “first root commit” you mean “the oldest commit on the branch’s first-parent chain”, the obvious local Git answer is:

That works because rev-list can walk parents locally. But your current problem is remote, over upload-pack, without a local repo. Git protocol v2 fetch gives you want, have, and optional filters; it does not expose a “tell me the first-parent root” query or a “first-parent only” traversal mode:

• gitprotocol-v2: https://git-scm.com/docs/gitprotocol-v2

So the answer splits into two cases.

If you control the source server There is a better way: add a tiny helper on the server side that runs native Git locally and returns either:

• the first-parent root commit for a tip, or
• a list of checkpoint commits along the first-parent chain.

That would be much better than downloading a commit-only graph into memory. Native Git can answer this cheaply, and if the server has commit-graph/bitmaps, it may be faster still. But this is a custom API, not something upload-pack gives you.

If you do not control the source server Then there probably is not a materially better exact method than what you have now. Your current approach in bootstrap.go fetches enough commit objects to reconstruct the chain client-side because the protocol does not provide the chain directly.

A few ideas that sound promising but mostly do not solve it:

• merge-base: wrong tool unless you are finding a common ancestor between two commits, not the root. Docs: https://git-scm.com/docs/git-merge-base
• Bitmaps: they can speed server-side pack construction, but they do not expose the first-parent root over the wire. Docs: https://git-scm.com/docs/bitmap-format
• Partial clone filters like tree:0: useful for reducing payload, but still not first-parent-only.

So my practical conclusion is:

• Without local disk and without a custom server endpoint, there is no obvious better exact way.
• If you can tolerate approximation, you can avoid the full chain and adaptively bisect on observed pack sizes.
• If you need the exact first-parent root or exact evenly spaced checkpoints, the clean solution is a source-side helper, not bitmap parsing.

If you want, I can sketch the smallest custom server API that would let replicate/bootstrap ask for first-parent checkpoints directly.

It means you stop trying to know the whole branch history in advance.

Instead of:

1. download commit graph
2. reconstruct full first-parent chain
3. choose exact checkpoints
4. fetch/push batches

you do something like:

1. try pushing the whole branch tip
2. if the pack is too large, pick some older ancestor as a smaller checkpoint
3. try again
4. if still too large, go further back
5. if comfortably small, move forward
6. repeat until you reach the tip

So “adaptively bisect” means: use success/failure and observed pack size to search for a commit that produces a pack under the target limit, roughly like binary search.

Example:

• target limit is 500 MB
• pack to tip looks like 2.4 GB -> too big
• try an older ancestor around “halfway back”
• that pack is 700 MB -> still too big
• try an even older one
• that pack is 320 MB -> good
• push that checkpoint
• now use it as a have and search for the next checkpoint

Why this is only an approximation:

• pack size is not proportional to commit count
• delta compression and shared objects make “half the history” not equal “half the bytes”
• so you learn by probing real packs instead of precomputing an exact chain model

The catch is important: you still need some way to name candidate older commits. Without the full first-parent chain, stock Git protocol does not give you an easy “jump to halfway ancestor” operation. So this idea only really works if you also have:

• a source-side helper API, or
• some cheaper way to discover ancestors, or
• a looser strategy that samples candidate commits rather than knowing the exact chain

So the short version is: “try a checkpoint, measure the real pack, and search forward/backward based on that,” instead of “compute all checkpoints up front from the whole chain.”

You wouldn’t, at least not exactly, with the current protocol surface.

That is the core constraint.

Without pulling enough commit data to walk parents yourself, or asking a custom source-side service, you do not have a way to say “give me a somewhat older ancestor of this tip” in any precise way. upload-pack lets you ask for objects by object id; it does not give you an ancestor-search API.

So if you remove the graph fetch, your options become:

• You already know some candidate older commits from somewhere else. For example: existing target refs, temp refs from prior batches, tags, cached checkpoints from an earlier run.
• You add a source-side helper. Then the server can answer “first-parent ancestor N steps back” or “checkpoint under X bytes”.
• You use a weaker heuristic that does not need exact ancestors. For example, only batch at known refs/tags. That is much less useful for a single long branch.
• You fetch commit objects incrementally anyway. At that point you are still rebuilding some amount of graph, just lazily instead of all at once.

So the honest answer is: without the graph, you generally do not know what an older ancestor is.

That’s why your current implementation pulls the commit-only graph in internal/strategy/bootstrap/bootstrap.go. It is paying that cost because Git’s normal remote protocol does not expose the ancestry navigation primitive you would need.

The practical takeaway is:

• If you need exact checkpoint commits in a no-disk client, you need commit graph data or equivalent commit data.
• If you want to avoid that cost, the clean alternative is not a smarter client algorithm; it is a smarter source server API.

The key question is really: “Can the client discover ancestors by itself without downloading commit ancestry?”

For stock Git protocol, the answer is basically no.

I’m checking GitHub’s official API/docs for commit ancestry and compare endpoints, then I’ll assess whether they can give the exact information your bootstrap planner needs.

Potentially yes, but only for GitHub-hosted repos, and with tradeoffs.

The promising parts are:

• The REST Git commit endpoint can return a commit object with its parents, and it can be used without auth for public repos or with Contents: read for private repos: https://docs.github.com/rest/git/commits
• The compare endpoint returns commits in chronological order and supports pagination: https://docs.github.com/en/rest/commits/commits?apiVersion=2022-11-28#compare-two-commits
• GitHub’s GraphQL API exposes commit history with pagination through Commit.history / CommitHistoryConnection: https://docs.github.com/en/graphql/reference/objects https://docs.github.com/en/graphql/guides/using-pagination-in-the-graphql-api

What that means for your use case:

• Exact root commit: yes, probably. You can walk parents[0] repeatedly via the Git commit API until you hit a commit with no parents. That gives you the first-parent root without downloading Git objects.
• Exact first-parent checkpoints: also yes in principle, by repeatedly following first parent and counting, or by using GraphQL commit history pagination if its ordering matches what you need. But this becomes many API calls on large histories.
• Better than pulling the graph: maybe for memory, yes; maybe for latency/rate limits, not obviously. For something Linux-sized, tens of thousands of first-parent steps through the API could be expensive and rate-limit-prone.

The main limitations:

• This only works for GitHub remotes, not generic Git servers.
• It depends on GitHub API auth/rate limits for large private repos; public repos may still be too slow if you walk commit-by-commit.
• The compare endpoint is useful when you already know both endpoints of a range; it does not by itself solve “find me the root.”
• I’m inferring the practical viability from the docs. The docs clearly expose parent links and paginated commit history, but they do not promise this as an efficient branch-root discovery primitive.

So the short answer is:

• For GitHub specifically, yes, the API can probably provide the information you need without pulling the Git graph.
• For a general git-sync design, no, because it would become GitHub-specific behavior and still may be too slow on very large histories.

The most credible GitHub-only strategy would be:

1. Detect a GitHub source.
2. Ask GitHub API for the branch tip commit.
3. Walk first-parent ancestry via commit parents.
4. Cache checkpoints aggressively.
5. Fall back to current graph-fetch logic for non-GitHub remotes or when API traversal is too large.

If you want, I can sketch the concrete GitHub-only algorithm and estimate how many API calls it would take for a large repo.

I’m looking at the local ../../git tree for protocol or transport-side primitives around ancestry, rev-list, bitmaps, or upload-pack that could expose something better than the current graph fetch.

There are a few protocol-side features worth checking more closely: shallow/deepen requests, object-info, and fetch --negotiate-only. I’m reading those codepaths now to see whether any of them expose ancestor discovery rather than just transfer negotiation.

The v2 docs confirm fetch has no ancestry query. I’m checking object-info now because if it exposes commit metadata over protocol, it could be a server-native way to walk parents without fetching packfiles.

object-info is a dead end for ancestry: in upstream Git it only exposes object size, not parents. I’m checking one more likely angle, shallow/deepen, because that is the only stock protocol feature that asks the server to compute ancestry boundaries.

I found one stock-Git mechanism that is closer than the current full-graph fetch: shallow/deepen can make upload-pack compute boundary commits for you. It’s not a first-parent oracle, but it is the only built-in server-side ancestry cutpoint primitive I’ve seen so far.

I checked ../../git, and the main result is:

There is no hidden stock protocol command that says “give me the first-parent root” or “give me the kth ancestor of this commit”.

The useful pieces are:

• fetch in protocol v2 supports want, have, filter, shallow/deepen, packfile-uris, etc., but no ancestry query in gitprotocol-v2.adoc.
• object-info exists, but upstream Git only supports object size, not commit parents or commit metadata: gitprotocol-v2.adoc protocol-caps.c
• --negotiate-only is only about finding commits you and the server already have in common, not arbitrary ancestors: fetch-options.adoc

The one trick that may actually matter is shallow/deepen.

Git’s own docs and code show that upload-pack can compute shallow boundaries server-side:

• deepen-since is internally like rev-list --max-age=...
• deepen-not is internally like rev-list --not <rev>
• the server replies with shallow <oid> boundary commits gitprotocol-capabilities.adoc upload-pack.c gitprotocol-v2.adoc

So the interesting idea is:

1. Request the branch tip with deepen N and filter tree:0.
2. Let the server send only the top N generations of commit objects.
3. Read the shallow-info response to get boundary commit IDs.
4. Parse only that partial commit set in memory.
5. Walk the first-parent chain within that partial slice.
6. If the root is not in the slice, deepen again.

Why this is interesting:

• It avoids fetching the entire commit graph up front.
• It still works without local disk.
• It uses only stock Git protocol features.

Why it is not a full solution:

• shallow depth is not first-parent depth; merges can produce multiple boundary commits.
• you still need to parse commit objects client-side for the fetched slice.
• upload-pack explicitly disables commit-graph for some shallow computations, so this may be slower server-side than a normal fetch path: upload-pack.c

So my current view is:

• No, upstream Git does not have a clean hidden “ancestor lookup” RPC you missed.
• Yes, shallow fetch is the closest stock trick if you want to reduce memory versus fetching the whole commit graph.
• It is best understood as incremental ancestry discovery, not direct root discovery.

If you want, I can sketch what a git-sync prototype would look like for:

• tree:0 + deepen N iterative root search, or
• a comparison of that approach against the current full-graph planner.

You still need some way to choose the next want.

After a retry, the target temp ref gives you the current base commit very cheaply:

• that becomes the have
• in your code that is current / ResumeHash for the temp ref in internal/strategy/bootstrap/bootstrap.go

But the graph fetch was never mainly about finding the base. It was mainly about finding the sequence of future checkpoint commits to use as wants:

• build first-parent chain
• place checkpoints
• on resume, find where the temp ref sits in that chain bootstrap.go

So:

• have: easy on retry, from the temp ref already on target
• want: still hard, unless you already know the next checkpoint

Cases where you do not need the graph again:

• The previous run already computed checkpoints and you persist/reuse them.
• The temp ref hash exactly matches one of the previously planned checkpoints, and you still have that checkpoint list.
• You switch to a different strategy that can discover the next checkpoint without the full chain.

Cases where you do still need graph/ancestry data:

• Fresh process, no cached checkpoint plan.
• --target-max-pack-bytes changed, so you want a different next checkpoint.
• The temp ref is stale and not aligned with prior checkpoints, and you need to replan from that point.

That is exactly why the current code has the “resume from stale temp ref by finding chain position” logic: if the temp ref is in the chain, it can replan remaining checkpoints from there, but it still needs the chain to do it internal/strategy/bootstrap/bootstrap.go.

So the short answer is:

You do not need the graph to recover the have on retry, but with the current algorithm you usually still need ancestry data to determine the next want, unless you cached the prior checkpoint plan or adopt a different checkpoint-discovery method.

Yes, for that shape of repo, it could help more than average.

For a Linux-style mirror with basically one dominant long branch:

• the current full-graph planner pays a very high upfront cost to learn one first-parent chain,
• but there is little branch fanout benefit from having the whole graph in memory,
• and retries are mostly about cutting one long history into acceptable chunks.

That makes an incremental shallow/deepen approach more plausible, because the problem becomes:

• find an acceptable checkpoint on one branch,
• push it,
• deepen or advance,
• repeat.

Why this helps more for “one long branch”:

• you are not repeatedly amortizing the graph fetch across many independent branches,
• the main value is discovering the next checkpoint on the same lineage,
• shallow boundaries are less noisy when the branch topology is simple.

Why it still does not magically solve everything:

• Linux still has many merges, so shallow depth is not equal to first-parent distance.
• A shallow boundary can include multiple commits, not “the one exact checkpoint you want”.
• If you care about clean first-parent batch spacing, you still need to inspect commit parents in each fetched slice.

But compared with your current approach, the trade looks better on Linux-like repos:

Current:

• one huge commit-graph fetch into memory
• extract first-parent chain
• compute all checkpoints

Possible alternative:

• fetch only a shallow slice of commit objects near tip with tree:0
• inspect that slice
• if it is still too large or not deep enough, deepen
• use the shallow boundary / parsed parents to choose the next checkpoint
• repeat

So for “mostly one long branch”, this can help because it turns:

• one massive up-front memory spike

into:

• repeated smaller ancestry probes

That is a real improvement if your biggest problem is memory/transient cost, not round trips.

The main tradeoff becomes:

• less peak memory
• more network round trips and more planner complexity

For Linux specifically, that may be worth it.

My best current read is:

• for many-branch repos, the current full-graph approach may amortize better;
• for single-dominant-branch repos like a kernel mirror, iterative shallow ancestry discovery is more attractive.

So yes: with that repo shape, this idea has a materially better chance of helping.

Yes. A few realistic examples:

• llvm/llvm-project
• microsoft/vscode
• kubernetes/kubernetes
• homebrew-core
• chromium mirrors, depending on how refs are exposed
• large internal monorepos with lots of release branches and automation refs

The important distinction is not just “many branches exist”, but:

• many branches are included in your replication scope
• and those branches have enough divergent history that planning one branch does not mostly solve the others

A repo can be huge and still behave like “one long branch” for your use case if you only mirror main. Conversely, a moderate-size repo can behave like “many branches” if you mirror lots of long-lived release branches.

For your batching question, homebrew-core is a good mental counterexample to Linux:

• lots of commits
• lots of branch/ref activity possible
• less of a single dominant lineage story

So yes, such repos definitely exist, but whether they matter depends on what refs git-sync replicate is actually asked to manage.

I’m checking how bootstrap planning is currently wired so I can expose it with minimal surface area and add a second planner strategy behind a flag instead of disturbing the execution path.

The clean place to expose this is the existing bootstrap command and unstable client, not plan. I’m tracing the config plumbing now so I can add a planner-only mode plus a planner strategy enum without affecting normal sync/replicate behavior.

I’m going to add this in two layers: first a planner-only bootstrap mode with visible checkpoint output, then a planner strategy enum so we can plug in a shallow planner behind the same surface. After that I’ll wire tests around the new non-push path.

I found a workable narrower prototype: planner-only plus an experimental shallow planner that only runs in planning mode and only for branch batching. It would use shallow, commit-only fetches to grow ancestry slices until the first-parent chain reaches the root, without ever pushing packs. I’m checking the existing chain walker now so I can tell whether that can be layered in cleanly.

I can make the experimental planner genuinely useful without touching the live push path: use increasing shallow tree:0 fetches from the tip, walk whatever first-parent prefix is present in that slice, and stop once the root is included. That gives us a real second planner to compare against the current full-graph planner.

I’m editing the bootstrap surface now: --plan-only plus --planner, a planner-only result shape, and no changes to live bootstrap execution unless the new flag is set.

I’ve got the planner split in; now I’m fixing the test doubles and wiring because the bootstrap source interface grew a depth-limited fetch method for the experimental planner.

I’ve finished the first pass of the API/CLI plumbing. I’m running focused tests now to catch compile breaks before I touch usage text and add coverage for the planner-only output.

There’s a straightforward compile collision: I introduced both a BootstrapPlan type and a BootstrapPlan function in syncer. I’m renaming the result type and rerunning tests.

The targeted packages built cleanly. I’m waiting on the full test run now, then I’ll add a small planner-mode test and tighten the help text.

I’ve added a CLI test for the non-push planner path and updated help text. I’m formatting the touched files and rerunning the relevant tests once more.

The CLI test change is in. I’m polling the focused test run now; if it passes I’ll finish with a quick usage example for the new planner flags.

Added a planner-only bootstrap path plus an experimental second planner strategy.

You can now run:

What changed:

• bootstrap now supports --plan-only and --planner graph|shallow.
• Planner-only mode computes bootstrap checkpoint plans without fetching relay packs or pushing to the target.
• Results include planner and bootstrap_plans in JSON when batch planning runs.
• The new shallow planner is experimental and only affects planning. It grows commit-only shallow slices (tree:0 + increasing deepen) until the first-parent chain reaches the root, so you can compare it against the existing full-graph planner without exercising the push path.

Notes:

• Normal bootstrap/sync/replicate execution is unchanged unless you explicitly use the new flags.
• The shallow planner currently depends on v2/filter/shallow support from the source and is aimed at batched branch planning, not the live push path yet.

Verification:

• go test ./... passed.