Can you take a look at the go code (wasm) in /Users/soph/Work/entire/devenv/entire-io-worktree1 based a bit on that I wonder if something like this can be build:

I’m going to inspect the wasm Go code in entire-io-worktree1 first to see what transport/protocol primitives already exist, then I’ll map that against what a remote-to-remote git sync would need.

The repo only has one Go/WASM entrypoint, so I’m narrowing to that path and the worker code that calls it. I want to see whether it already parses packfiles/commits or if it only hits GitHub’s HTTP APIs.

The Go side is already speaking smart HTTP upload-pack directly and parsing packfiles in memory. Next I’m checking the TypeScript wrapper to see how much of the transport loop is already there and where the gaps are for a remote-to-remote mirror.

The existing code proves the read side: it can build upload-pack requests, fetch packfiles over smart HTTP, and parse them entirely in memory. The missing question is the write side: whether there’s any equivalent receive-pack/push support or pack generation already present.

Yes. The code in api/wasm/git-graph.go and api/src/lib/git-graph.ts already proves the hard half: a repo can be spoken to over smart HTTP directly, with no .git checkout, by building upload-pack requests in WASM, fetching packfiles, and parsing them entirely in memory. It already does:

• refs discovery via /info/refs?service=git-upload-pack
• pkt-line request construction in WASM
• packfile parsing into an in-memory object store
• object/tree walking and diffing without a local repo

What it does not have is the push half. I found no receive-pack support, no push/update request builder, and no packfile encoder for writing objects back out. Right now it is a read-side transport/parser, not a mirror engine.

So: conceptually, a git sync <source> <target> built only from git protocol primitives is feasible. But the real design is not “fetch from A and forward bytes to B”; it is:

1. ls-refs/advertise refs from target
2. ls-refs/advertise refs from source
3. Compute which refs should move
4. Ask source for the object graph needed for those target updates
5. Materialize those objects in memory
6. Build a push pack for target
7. Send receive-pack update commands + packfile to target
8. Parse target status and retry/idempotently continue later

The main constraint is this: “just stream blobs from A to B” is too optimistic. Git push is object-graph based, not blob-forwarding based. In practice you need commits, trees, tags, and blobs, and the target cares about its own existing haves. The pack you get back from source upload-pack is also not directly the thing you POST to target receive-pack; you usually need to repackage or at least reframe it.

The code you have is a strong base for an MVP because it already has:

• in-memory pack parsing: parsePackToStore
• upload request building: buildRequest
• TS transport loop around smart HTTP: fetchBranchCommitGraph

What you’d need to add:

• receive-pack ref advertisement parsing
• push/update request builder for create/update/delete refs
• packfile encoder from the in-memory object store
• negotiation against target refs so repeated syncs only push missing objects
• ref mapping rules like refs/heads/* -> refs/heads/*, tags, prune policy, force policy

My recommendation is an MVP with these limits:

• branch mirrors only, no tags initially
• fast-forward only
• no prune
• full object closure for changed refs
• in-memory object store only, spilling to temp files only if packs exceed memory limits

That would already give you a rerunnable remote-to-remote mirror tool with no local repository checkout. If you want, I can sketch the exact protocol flow and a minimal package layout for implementing it in this git-sync repo.

I’m going to inspect this repo’s current structure and then wire an MVP around the existing read-side protocol approach: source refs, source pack fetch, target receive-pack push, fast-forward branch updates only.

The repo is effectively empty, so I’m setting the MVP shape from scratch rather than integrating into existing code. Next I’m checking local tooling and then I’ll scaffold a Go CLI that can talk smart HTTP directly.

I have Go available and an empty repo. I’m checking whether go-git is already present in the module cache; if it is, I can build the MVP without a dependency download round-trip.

The module cache already has go-git and go-billy, so I can keep moving locally. I’m going to inspect the available push/transport APIs in that cached version before I write the CLI around them.

Plan:

1. Scaffold a Go CLI module with a sync command and token/header auth support.
2. Implement remote branch discovery, in-memory fetch from source, and fast-forward checks against target.
3. Push changed branches to target over smart HTTP using the in-memory object store.
4. Add basic tests for ref selection/fast-forward logic and write a minimal README with usage and constraints.

I’ve confirmed the simplest MVP path: use go-git’s transport and pack primitives end to end, with a repository backed only by memory.NewStorage(). That still avoids any local checkout, while letting the library handle receive-pack and pack writing correctly.

I’ve got the implementation shape. I’m creating a small Go module now with a real sync command, plus a separate planning layer so the fast-forward rules are testable without network access.

The first pass is in place. I’m compiling now to flush out API mismatches and trim anything I added that the compiler doesn’t actually need.

The code is structurally there; the current failure is just module metadata. I’m generating go.sum next, then I’ll rerun tests and fix any actual compile/runtime issues that surface.

The remaining blocker is dependency resolution and build cache location, not the code itself. go mod tidy needs writable module/build cache state and at least one dependency fetch, so I’m requesting that directly rather than trying to work around it.

Dependency resolution is done. I’m rerunning the test/build loop with cache directories redirected into writable space so I can get a clean compile signal.

I hit the expected second layer now: compile-level cleanup and a test command that should run fine once the code trims unused pieces. I’m fixing those imports and small logic rough edges before rerunning go test.

One test failure exposed a real issue in the fixture, not the planner: my synthetic commits were hashing identically because their contents were identical. I’m making those test commits unique and rerunning the suite.

The planner and unit tests are green now. I’m doing one final build pass and then I’ll summarize what the MVP can and cannot do yet.

Built the MVP in this repo. The CLI lives at cmd/git-sync/main.go, the sync engine is in internal/syncer/syncer.go, and usage/limits are documented in README.md.

The command is:

with --branch, --dry-run, and separate source/target auth flags or env vars. The MVP does branch-only mirroring over smart HTTP, uses an in-memory repo only, creates missing target branches, fast-forwards existing ones, and blocks non-fast-forward/diverged targets.

Verification:

• env GOCACHE=/tmp/go-build go test ./... passed
• env GOCACHE=/tmp/go-build go build ./cmd/git-sync passed

Two important limits:

• I have not run an end-to-end sync against real remotes yet.
• This uses go-git’s transport/push machinery rather than a custom handwritten pkt-line/pack encoder, but it still avoids a local checkout and stays in-memory.

If you want, next I can add an integration test path and a real dry-run/status output that shows source and target SHAs per branch.

Right now it is a small CLI that mirrors branch refs from one remote to another without creating a local checkout.

Flow:

1. It creates an in-memory Git repository.
2. It connects to the source remote and lists refs.
3. It connects to the target remote and lists refs.
4. It decides which branches to consider.
5. It fetches the selected source branches into in-memory remote-tracking refs like refs/remotes/source/main.
6. It fetches the selected target branches into in-memory remote-tracking refs like refs/remotes/target/main.
7. For each branch, it compares source and target tips:
   • if target branch does not exist: plan create
   • if target tip equals source tip: plan skip
   • if target tip is an ancestor of source tip: plan update
   • otherwise: plan block
8. If not --dry-run, it pushes only the create and update branches to the target.

The core logic is in internal/syncer/syncer.go. The CLI parsing is in cmd/git-sync/main.go.

What it does not do yet:

• no tags
• no prune/delete
• no force push
• no custom ref mapping
• no persistent state between runs
• no custom low-level pack/protocol implementation; it uses go-git transport/push internals

So this is already usable as a safe MVP for “mirror branches from A to B if B is empty, equal, or behind A”. It intentionally refuses diverged branches.

git fetch asks another repository for refs and objects, downloads anything you are missing, and updates your remote-tracking refs. It does not change your current branch or working tree.

In practice it usually does this:

1. Contact the remote and read its advertised refs like refs/heads/main.
2. Compare those refs with what your local repo already has.
3. Negotiate which objects you already have and which commits/trees/blobs you need.
4. Download a packfile containing the missing objects.
5. Store those objects in .git/objects.
6. Update refs like refs/remotes/origin/main.

What it does not do:

• it does not merge
• it does not rebase
• it usually does not touch files in your checkout

That is why git pull is basically:

1. git fetch
2. then merge or rebase onto what was fetched

So the important idea is: fetch is “synchronize object database + remote-tracking refs”, not “update my branch contents on disk.”

Yes: git push.

Roughly:

• git fetch: remote -> local
• git push: local -> remote

fetch downloads objects and updates your local view of remote refs. push uploads objects and asks the remote to move its refs.

They are not perfectly symmetric, though:

• fetch is usually safer and more permissive
• push can be rejected for non-fast-forward updates, permissions, branch protection, hooks, and policy

At the protocol level the pair is:

• git-upload-pack for fetch
• git-receive-pack for push

So for your tool, the “opposite of fetch” on the target side is effectively a push.

In the ideal case: only the new commit and any objects reachable from it that the target does not already have.

For one extra commit on top of a shared branch tip, that usually means:

• the new commit object
• any new tree objects for directories changed by that commit
• any new or changed blob objects for files changed by that commit
• ref update metadata

If the commit changes a tiny file, the transfer can be very small. If it adds a 50 MB blob, you transfer that blob.

For the current MVP specifically, the flow is less optimal than a hand-tuned protocol bridge:

1. It fetches the selected branch from source into the in-memory repo.
2. It also fetches the selected branch from target into the same in-memory repo.
3. Then it pushes from memory to target.

So total network for that branch is roughly:

• source -> tool: source branch objects needed to materialize the source tip
• target -> tool: target branch objects needed to materialize the target tip
• tool -> target: only the missing objects needed for the push, plus the ref update

If source and target already share almost everything and source only added one small commit, then:

• source fetch should be close to “just that new commit and its changed objects”
• target fetch may still pull enough commit history to determine ancestry
• push should be close to “just that new commit and its changed objects”

So the push side is incremental, but the current fetch side is not yet aggressively minimized. A more advanced version could reduce this further by negotiating directly against target’s advertised haves before fetching from source.

Yes. The simplest useful version is to fetch the target branches into memory first, then fetch the source branches into that same object store. That lets the source-side fetch negotiate against objects we already learned from the target, so for the “one new commit on top” case the source transfer can collapse much closer to just the delta.

I’m also updating the docs to reflect the new transfer behavior so the repo matches what the command now does.

This change is small but meaningful, so I’m rerunning formatting and tests now.

Yes, and I added the first practical version of it.

What changed:

• In internal/syncer/syncer.go, the tool now fetches the target branches first, then fetches the source branches into the same in-memory object store.
• That means the source fetch can advertise “have” objects that came from the target, so shared history should not be re-downloaded from the source.

Effect on your example:

• If target already has branch tip A
• source has A -> B
• then after fetching target first, the source fetch can usually negotiate from A and transfer mostly just B and any new trees/blobs reachable only from B

What this improves:

• source -> tool transfer becomes much closer to incremental
• push to target was already incremental

What it does not do yet:

• it still fetches target objects into memory first, so there is still target -> tool traffic
• it is not yet a true pack relay from source to target
• it still relies on go-git negotiation behavior rather than custom low-level upload-pack orchestration

Docs were updated in README.md, and env GOCACHE=/tmp/go-build go test ./... still passes.

The next step, if you want to push this further, is to stop fetching full target branch objects when we only need target tip commits/ancestry checks. That would reduce target -> tool traffic too.

I’m going to inspect go-git’s server-side test helpers and transport packages, then I’ll build an integration harness around two in-process HTTP servers so we can assert both correctness and transfer size.

I found the go-git server primitives and the HTTP test suite. Next I’m reading their in-process HTTP server setup so I can reuse the same pattern instead of inventing a new protocol shim.

The built-in HTTP transport tests use git-http-backend, but go-git itself has server-side upload-pack/receive-pack sessions. I’m looking at the HTTP client expectations now so I can expose those sessions over httptest and keep the whole test stack inside Go.

I have enough of the server pieces now. The remaining bit is the smart-HTTP framing: info/refs responses and the two service POST endpoints. I’m checking whether go-git already exposes the exact encoding helpers; otherwise I’ll add a tiny test-only handler that wraps the server sessions correctly.

I’m wiring two integration cases now: initial sync to an empty target, and a resync after one extra source commit. I’m also fixing the target-fetch path to tolerate an empty target repo, because the tests should cover that baseline behavior too.

The integration harness is in. I’m compiling and running the new tests now; I expect a couple of small fixes around imports or smart-HTTP framing before the full loop passes.

The first failure is just the expected missing object import in the test helper. I’m fixing that and rerunning the integration cases.

The integration harness needs to open local loopback listeners via httptest, and the sandbox blocked that. I’m requesting permission to run the integration test process outside the sandbox so I can validate the HTTP server path end to end.

The first real protocol issue showed up: Remote.ListContext against an empty target returns ErrEmptyRemoteRepository. Since we don’t actually need target refs from ListContext for planning, I’m removing that failure path and relying on the target fetch phase instead.

The resync result is useful: it shows that “smaller pack bytes” is not a stable assertion against go-git’s server implementation, even when the negotiation path is correct. I’m switching the integration check to something more robust: verifying that the second source fetch actually sends have lines, which proves it is negotiating against objects learned from the target.

I replaced the brittle “pack must be smaller” assertion with a protocol-level check: the resync source fetch must include have lines. That directly tests the optimization we added.

The test exposed a real gap: fetching target first does not, by itself, make go-git send have lines on the subsequent source fetch. I’m inspecting go-git’s fetch negotiation now to see what it considers eligible local refs, because that determines whether we can make this optimization work or need a lower-level fetch path.

I found the issue with the assertion, and it’s in the test harness rather than the sync logic: packp.UploadPackRequest.Decode does not populate the have list. I’m switching the metric to count have lines directly from the raw pkt-line request body.

The integration tests are passing now. I’m doing one final full test pass across the repo so the new harness and the existing unit tests are green together.

Added end-to-end integration coverage in internal/syncer/integration_test.go and fixed empty-target handling in internal/syncer/syncer.go.

The integration harness now:

• starts source and target as in-process smart-HTTP servers backed by go-git
• uses real upload-pack and receive-pack flows
• records request/response byte counts per service
• inspects the source upload-pack request body to verify have lines are sent on resync

What the tests validate now:

• initial sync to an empty target works
• target ends up at the exact same branch tip as source
• resync after one extra source commit performs a real push/update
• the second source fetch advertises have lines, proving it is negotiating against objects learned from the target

Verification:

• env GOCACHE=/tmp/go-build go test ./... passed

One important outcome from the integration work: the robust thing to assert is negotiation behavior (have lines), not “pack bytes must be smaller”. With go-git’s server implementation, the byte-size assertion was not stable enough.

If you want to continue, the next step is to reduce target-side transfer, since the current tool still fetches target history into memory on every run before it can negotiate source fetches.

I’ve validated the current negotiation improvement. The next change is architectural: stop fetching target objects at all, and instead use target ref tips as haves in a direct source-side upload-pack request. Then we can do fast-forward checks from the fetched source commit graph without materializing the target history locally.

I’ve got the path: remove the target fetch entirely, list target refs via receive-pack, fetch source with explicit haves derived from target branch tips, and switch the fast-forward check to walk only the fetched source-side commit graph. Then the integration tests should show zero target upload-pack traffic.

I hit one expected compile cleanup from that refactor: the commit-walk helper needs the repository storer passed in explicitly because object.Commit doesn’t expose it. I’m fixing that plus any remaining import/API mismatches now.

The integration path is green with the new flow. I’m tightening the test expectations slightly so they explicitly prove target upload-pack traffic is gone, then I’ll run the full suite.

Changed the sync flow so it no longer fetches target objects at all.

What it does now:

• lists source branches via upload-pack refs
• lists target refs via receive-pack refs
• fetches from source using:
  • want = source branch tips
  • have = current target branch tips
• stores only the source-side pack in memory
• checks fast-forward by walking the fetched source commit graph until it reaches the target hash
• pushes to target with the same in-memory source objects

So target-side transfer is now reduced to:

• receive-pack ref advertisement
• the final push

There is no target upload-pack POST anymore.

The main code change is in internal/syncer/syncer.go. The integration tests in internal/syncer/integration_test.go now explicitly assert:

• initial sync to empty target works
• resync sends have lines to source
• target receives no upload-pack requests

Verification:

• env GOCACHE=/tmp/go-build go test ./... passed

This gets much closer to the design you originally wanted: the tool now uses target refs as negotiation input, without materializing target history locally. The next step, if you want to keep pushing, is to stop using go-git’s high-level push path too and build the source-fetch/target-push bridge entirely from lower-level protocol calls.

I’m replacing the remaining high-level push path with direct receive-pack usage: build the update commands ourselves, compute the object closure from the in-memory store, encode a packfile, and send a ReferenceUpdateRequest straight to the target.

I’ve confirmed the direct push ingredients exist in go-git: low-level ReferenceUpdateRequest, revlist.Objects, and packfile.NewEncoder. I’m using those now so the tool no longer depends on Remote.PushContext.

I’m swapping out Remote.PushContext now. The new path will build packp.Commands directly from the plans, compute the object closure with revlist.Objects, pack those objects with packfile.NewEncoder, and send the request through a raw ReceivePackSession.

The direct push path is in. I’m running the suite now to catch any compile issues or protocol regressions from removing Remote.PushContext.

Two compile fixes surfaced: I still need config.RemoteConfig for the source remote, and the generic storer doesn’t expose Config(). I’m restoring the config import and using a fixed pack window for the direct encoder, matching go-git’s server default.

The tool now does both sides with lower-level protocol/session calls.

What changed in internal/syncer/syncer.go:

• source fetch is already direct upload-pack
• target push is now direct receive-pack
• Remote.PushContext is gone
• push requests are built as raw packp.Commands
• object closure is computed with revlist.Objects
• the outgoing pack is encoded with packfile.NewEncoder and streamed into ReferenceUpdateRequest.Packfile

So the end-to-end flow is now:

1. list source refs
2. list target refs
3. fetch source with want=source tips and have=target tips
4. fast-forward check by walking fetched source commits
5. build raw receive-pack update commands
6. stream only the needed objects to target

The integration tests still pass, including the assertions that:

• resync source fetch sends have lines
• target receives no upload-pack requests
• sync still completes correctly

Verification:

• env GOCACHE=/tmp/go-build go test ./... passed

At this point the main remaining high-level bit is source ref listing via Remote.ListContext. If you want, the next step is to replace even that with direct smart-HTTP ref advertisement handling so the whole tool is consistently low-level.

Yes, but smaller benefits than the changes we just made.

What you already gained from the recent work:

• no target fetch pack at all
• direct control over source have negotiation
• direct control over target push commands and pack generation
• cleaner observability for what is actually transferred

Replacing source ref listing too would give:

• one consistent protocol layer everywhere
• easier instrumentation of exact bytes and capabilities on every request
• easier future support for custom negotiation/ref filtering
• fewer hidden go-git policy decisions in the control path

What it would not materially improve:

• transfer size, in most cases
• runtime, by much
• correctness, unless go-git list behavior becomes limiting

So the answer is: yes, but mostly architectural benefits, not a big performance win.

If the goal is practical payoff, I would prioritize one of these next instead:

• explicit branch mapping and mirror policies
• tags support
• protocol/byte stats output for real runs
• force/prune modes with safety rules
• protocol v2 support if the servers benefit from it

If the goal is purity and full control of the transport, then replacing ref listing is the next obvious step.

I’m going to replace source ref listing with direct session-based advertisement handling first, then I’ll layer in the next set of features in the least disruptive order: stats output, branch mapping, tags, force/prune controls, and protocol v2 detection/support where the transport exposes it.

I’m folding the remaining feature work into the same pass. Some of it is straightforward CLI/planning work; protocol v2 is the one likely to land as detection/reporting rather than full use, because go-git’s HTTP transport in this version doesn’t expose a v2 request path.

I’m doing a larger rewrite of the sync engine now. That’s the pragmatic way to land the remaining features coherently instead of layering more special cases onto the current file.

I’ve replaced the sync engine core. The next pass is compiler cleanup plus compatibility shims for the existing unit tests, then I’ll wire the new CLI flags and add focused tests for mapping/tags/prune/force.

I’m adding integration coverage for the new planning features now: mapping, tags, prune, and force. That will give us a clean signal before I update the docs.

I’m finishing the low-level ref listing work and wiring the remaining features through tests and docs. First step is to format the touched files and run the full suite so I can fix whatever broke in the combined change set.

gofmt hit README.md, so I’m re-running it on Go files only and waiting for the test failures to settle. After that I’ll fix compile or behavior regressions in one pass.

The first failure is in the new tag/prune path: we’re trying to build a push pack for a tag object we never fetched. I’m reading the sync planner and fetch set now, then I’ll patch the fetch closure so tag updates have the objects they need.

I found two logic gaps from the new feature batch. First, tag creates can point at objects the target already has, so the push path must not insist those objects exist locally. Second, prune currently never sees extra target tags, so it can’t delete them; I’m fixing both now, and I’ll clean up the stats double-count on info/refs in the same patch.

The fetch/push boundary is patched so ref-only retargets can avoid requiring local copies of already-present target objects, and prune now considers unmanaged target tags. I’m reformatting and rerunning the suite to see what the next failure is.

The receive-pack failure looks like a protocol capability issue rather than an object issue now. I’m checking the push request capabilities and command mix next, especially delete handling, because prune is the first path that sends deletions.

go-git’s helper only seeds agent and report-status; it does not actually enable delete-refs despite the comment. I’m making deletion explicit in the push request now, which should clear the prune path if that’s the server-side rejection.

I’ve enabled delete-refs explicitly on prune pushes. Running the targeted integration test again now to see whether that was the protocol mismatch or whether there’s another receive-pack edge behind it.

The in-process HTTP test server is slightly wrong here: go-git’s receive-pack can return a report plus a command error, and the smart-HTTP layer is supposed to return that report body with 200, not collapse it into 500. I’m fixing the harness so the failing command status comes back properly, which should tell us whether the remaining issue is real sync logic or just the test server behavior.

The harness now returns receive-pack status reports instead of masking them behind HTTP 500s. Re-running the targeted test should expose the real protocol or command-level rejection if one is still there.

The remaining failure is real protocol behavior: this server expects a pack stream whenever the request contains create or update commands, even if every referenced object already exists on the target. I’m changing the push path to send an explicit zero-object pack in that case instead of omitting the pack section.

The push path now distinguishes “no pack needed because this is delete-only” from “create/update commands with zero new objects,” and it will stream an explicit empty pack for the latter. I’m rerunning the tag/prune/force integration case to verify that edge is covered.

The next mismatch is behavioral: blocked non-fast-forward refs currently produce a plan entry but not an error, which makes --force less meaningful than it should be. I’m tightening that so a real sync run aborts on blocked refs before it pushes anything, while --dry-run can still report the block without mutating the target.

Blocked refs now stop a real sync run before any push happens, which matches the intended safety boundary for the new force mode. I’m rerunning the force/prune test first, then the full suite if that clears.

The force/prune path is green now. I’m running the full suite next, then I’ll bring the README in line with the new low-level ref listing, stats, mapping, tag, prune, and force behavior.

The code is green. I’m just bringing the README up to date now so it matches the current transport design and the new flags rather than the original branch-only MVP description.

The README is aligned with the current implementation now. I’m doing one quick status check so I can summarize the exact outcome and any remaining gap cleanly.

Implemented the low-level ref listing and finished the next feature batch. Source and target ref discovery now go through direct info/refs handling in internal/syncer/syncer.go, so the sync path is consistently low-level on list, fetch, and push.

The open items covered in this pass are in place:

• transfer stats via --stats
• explicit ref mapping via --map
• tag mirroring via --tags
• force and prune safety via --force and --prune
• protocol flag plumbing via --protocol auto|v1

The CLI wiring is in cmd/git-sync/main.go, the new integration coverage is in internal/syncer/integration_test.go, and the updated behavior/docs are in README.md.

One remaining gap is protocol v2 itself: the flag exists, but auto currently resolves to v1 and the README calls that out explicitly in README.md.

Verification: env GOCACHE=/tmp/go-build go test ./... passed.