We have a repo with 50k refs and we go this error: 2026/06/18 13:17:39 WARN Permanent sync error, terminating error="sync: replicate relay failed: replicate execute: push target refs: target receive-p

The error string isn't in git-sync — it's enforced server-side. The server is the entire backend. Let me search there while the agent examines the push path.

Done. Here's a summary.

Root cause

The error is the server (entiredb) rejecting a push: receive_pack.go:53 caps a single receive-pack request at maxRefUpdateCommands = 25_000 (rejection at count > limit). Your repo's relay sync sent all 55,006 ref updates in one request because git-sync's relay strategies send every command at once with no count-based batching:

• replicate.go:70/78, incremental.go:66/85, materialized.go:123 → gitproto.PushPack / PushCommands / PushObjects, each one request.

(Only the bootstrap strategy batched, and only by pack size, not command count.)

Fix — client-side batching in internal/gitproto/push.go

Centralized in the three push primitives so all four strategies are covered:

• New maxRefUpdatesPerPush = 20_000 (headroom under the 25k cap) + chunkRefUpdates helper.
• PushPack: full pack rides with the first ≤20k commands; remaining refs follow as ref-only PushCommands batches.
• PushCommands: loops over chunks (extracted pushCommandsBatch).
• PushObjects: materialized pack rides with the first update batch; remaining updates and any deletes go ref-only (extracted pushObjectsBatch).

The split is safe because the first pack carries all objects, and receive-pack commits the entire received pack — verified at both ends: entiredb's CommitQuarantinedFanout ("we commit even when some refs failed connectivity… the pack may carry objects shared with refs that did pass") and canonical git's tmp_objdir_migrate (migrates everything, no reachability pruning). So later ref-only batches resolve against objects already on the server.

Added tests (TestChunkRefUpdates, TestPushCommandsBatchesOverCap, TestPushPackBatchesOverCap). Full suite + go vet pass.

Does it work with GitHub? Yes.

• GitHub's "limit branches/tags per push" is opt-in (default: no limit), so 20k-ref batches are fine on default repos. (push policy docs)
• The "full pack first, ref-only rest" design relies on quarantine migration installing the whole pack, which is exactly how canonical git behaves (quarantine commit, tmp_objdir_migrate). Unreferenced objects persist (only GC reclaims them, with grace periods — not triggered between two pushes seconds apart).
• GitHub's other relevant limit is the 2 GiB push size, already handled separately by pack-size limiting.

Caveats worth noting: (1) if a target repo explicitly enables the per-push branch/tag limit with a small value, no count-batching short of that value would satisfy it — that's a deliberate restriction, not something to design around. (2) If a sync is interrupted between batches, the target holds the objects (unreachable) plus a partial ref set; the next run re-plans and completes it.

I did not touch the server limit — client batching respects the cap and works against any target, including GitHub.

Sources: push policy, 2 GiB limit, git quarantine, tmp-objdir.c

The four cleanup agents are running concurrently. I'll wait for all of them to report, then dedup and apply fixes.

I have the simplification findings. Waiting for the reuse, efficiency, and altitude agents to complete before I dedup and apply.

I have two of four. Waiting for efficiency and altitude before applying.

Three of four in. Waiting for the efficiency agent before applying the deduped fixes.

All green after the refactor. Here's the /simplify summary.

Applied

Extracted splitFirstBatch helper (internal/gitproto/push.go) and used it in both PushPack and PushObjects, replacing the duplicated

This was the consensus finding across three of four agents (reuse, simplification, altitude) — one DRY fix that also resolves the efficiency agent's nil-slice notes (the helper returns nil for rest in the common case). Build, vet, and the gitproto + strategy test suites all pass.

Skipped (with reasons)

• Altitude / constant placement — agents confirmed batching at the gitproto push-primitive layer is the correct depth (all strategies funnel through these functions; a hardcoded maxRefUpdatesPerPush is right since it mirrors a server limit, not a user tunable). Nothing to change.
• chunkRefUpdates single-element wrapper for small input (efficiency #3) — intentional: it keeps the caller loop uniform and preserves one-request behavior for the empty slice. The allocation is a stack-level slice header, negligible. Skipped as a false positive.
• PushObjects updates/deletes partition allocation (efficiency #4) — the efficiency agent itself concluded it's unavoidable and clean (deletes must go ref-only; only runs on >cap pushes). Skipped.
• Test helper pushRecorder vs existing captureReceivePackBody (reuse #2) — both the reuse and the reviewing agents judged it acceptable; captureReceivePackBody is single-request/channel-based and can't record multiple batches. Genuinely different purpose. Skipped.
• Missing explicit PushObjects batching test (altitude note) — a coverage gap, not a simplification/quality issue in the changed code, and a real PushObjects test needs a populated object store + delta selection. Out of scope for /simplify; worth a follow-up if you want it.

Done. PR opened: https://github.com/entireio/git-sync/pull/94

• Branched off main → fix/batch-ref-updates-receive-pack
• Committed the batching change + tests (2 files, +234/−2)
• Pushed and opened PR #94 with a writeup covering the root cause, the fix, the GitHub-compatibility verification, and test coverage.

The fixed binary is at /tmp/git-sync (built from this branch).

1. Set a GitHub PAT (needs repo scope — read on source, write on target):

2. Preview first (no pushes) — confirms it sees all ~55k refs:

3. Run the replicate (this is the path that previously failed at 55,006 > 25,000):

Notes:

• --all-refs mirrors every refs/* (branches, tags, notes, pulls, custom namespaces) — needed to actually push all 55k. Without it, replicate defaults to just branches.
• Args are positional <source-url> <target-url>; the env vars supply auth (or use --source-token/--target-token). GitHub accepts the PAT as the password with the default git username.
• With the fix, the 55k refs push as ~3 batches of ≤20k (first carries the pack, rest are ref-only) instead of one 55,006-command request.
• plan against an empty target will show all refs as creates; if the target already has some refs from a prior attempt, it'll show only the diff.

To test against the old behavior for comparison, you'd build from main (git stash not needed — just git checkout main && go build -o /tmp/git-sync-old ./cmd/git-sync) and run the same replicate — it should fail with the too many ref-update commands error.

Rebuilt /tmp/git-sync with a tunable batch size. Now retry with a small batch to confirm GitHub is the bottleneck — start conservative:

(Use the freshly built /tmp/git-sync, not go run — though go run from this branch would also pick up the change.)

What this tells us:

• If it succeeds at 2000 → confirmed: GitHub can't handle 20k ref updates per push; the fix is correct but the default batch is too big for GitHub. We then decide on a sensible default / per-target flag.
• If it still 500s at 2000 → the cause isn't batch size (could be GitHub disliking 55k total refs in one repo, or something else), and I'll dig further.

You can sweep upward (5000, 10000) to find GitHub's actual ceiling so we pick a good default.

Two notes:

• The pack push itself worked, so this is purely about ref-update batch size.
• I've not committed this env-var change yet — it's a diagnostic knob for now. Once we know GitHub's threshold, I'll fold it into the PR properly (likely a --target-max-ref-updates flag plus a safer default, rather than only an env var). Want me to add the flag now, or wait for your test result?

Given this is GitHub-specific behavior worth recording, do you want me to also note the finding (GitHub 500s on large ref-update pushes) in the PR description?

That's exactly the signal we wanted — it's past the first batch that previously 500'd. At 2,000 refs/batch GitHub is accepting the pushes, so the GitHub 500 was indeed caused by too many ref updates in one request (20k was too big; 2k works). It's now grinding through ~28 batches (55,008 ÷ 2,000), each a ref-only follow-up push.

The target: target: target: ... is cosmetic, not an error. Here's what's happening:

• The first batch carried the pack → you saw "Resolving deltas: 100% ... done".
• Each subsequent ref-only batch opens its own sideband progress stream. GitHub emits an empty/near-empty progress message per batch, and our progress writer prints the "target: " prefix for it with no payload behind it — so you get one bare prefix per batch (~27 of them for the remaining refs).

Let it finish — if it completes without an error line and exits 0, the mirror succeeded and all 55,008 refs are on the target. You can verify after with:

Findings confirmed:

1. ✅ The batching fix bypasses entire-server's 25k cap (the original bug).
2. ✅ GitHub has a much lower practical ceiling for ref updates per push — 20k → 500, 2k → fine. Worth finding the real ceiling (try 5000, 10000) so we pick a good default.

Two things I'll fold into the PR once you confirm completion:

• A safer default and/or a --target-max-ref-updates flag (keeping the env var as an escape hatch) — 20k is fine for entire-server but clearly too high for GitHub.
• A tiny fix to suppress the empty "target: " progress prefix on ref-only batches so the output isn't noisy.

Want me to start a batch-size sweep recommendation now, or wait until you've found GitHub's ceiling? And should I add the GitHub-500 finding to the PR description?

Git has no native wildcard delete — you can't do git push origin :refs/heads/*. You have to enumerate the refs and push deletes (the :refname refspec means "delete"). And since GitHub 500s on large ref batches, the deletes need the same batching.

For a throwaway test repo, the cleanest reset is just to recreate it:

That gives you a clean empty target instantly and sidesteps the delete-batching problem entirely.

If you'd rather delete refs in place, enumerate with ls-remote and batch the deletes with xargs (2000 per push, since that worked):

How it works:

• git ls-remote "$URL" 'refs/heads/*' 'refs/tags/*' lists matching refs as <sha>\t<refname>.
• awk '{print ":" $2}' turns each into a delete refspec :refs/heads/foo.
• xargs -n 2000 deletes 2000 at a time, keeping each push under GitHub's ceiling.

Notes:

• Don't include refs/pull/* — those are GitHub's read-only PR refs and deletes will fail. Stick to the namespaces you actually synced (heads, tags, plus any custom ones from --all-refs).
• If you want everything except pull refs in one go: git ls-remote "$URL" | grep -v $'\trefs/pull/' | awk '{print ":" $2}' | xargs -n 2000 git push "$URL".

Given it's a test repo, I'd just gh repo delete/create — faster and guaranteed clean for your 10k run.

Done. Pushed and PR #94 updated.

What changed:

• Default maxRefUpdatesPerPush → 5,000 (safe for GitHub and conservative providers).
• GITSYNC_MAX_REF_UPDATES_PER_PUSH env var raises it for the mirror-pipeline / entire-server targets (up to their 25k cap) to cut round trips. Invalid/non-positive → falls back to default.
• Added TestResolveMaxRefUpdatesPerPush for the override + fallback.
• PR description now documents the GitHub-500 finding with the batch-size table (20k/10k ❌ on GitHub, 5k ✅; all fine on entire-server).

Build, vet, full suite pass; and you've confirmed it live (55,008-ref GitHub→GitHub mirror works at 5k).

For the mirror-pipeline, set GITSYNC_MAX_REF_UPDATES_PER_PUSH=20000 (or up to 25000) in its environment when the target is entire-server.

Two optional follow-ups still on the table, say the word and I'll add them to the PR:

1. --target-max-ref-updates CLI flag in addition to the env var, for discoverability/per-invocation control.
2. Suppress the empty target: progress prefix on ref-only batches (the cosmetic line-spam you saw) — more visible now that the default is 5k (≈11 batches for 55k refs).

That's not a git-sync issue — it's GitHub Push Protection (secret scanning) on the target repo rejecting the refs/entire/checkpoints/* refs. Note the message: "Timed out validating Secrets detected, please try again" → GitHub's secret scanner is timing out validating those checkpoint commits, and push protection (GH013) declines the push.

Two things are colliding:

1. The source's refs/entire/checkpoints/* are Entire's checkpoint/session refs — snapshots of working state, exactly the kind of content secret scanning flags.
2. You're pushing a huge number of them, so GitHub's per-push secret validation times out rather than cleanly passing/failing.

(This earlier succeeded at 5k presumably because the freshly created repo didn't have push protection on yet, or the scanner didn't time out that run. Recreated repos pick up your org's default security settings.)

You don't need to change git-sync. Pick one:

Option A — disable push protection on the target (keeps all 55k refs, best for your batching test):

(or GitHub UI: Settings → Code security → Push protection → off)

Option B — don't mirror the checkpoint namespace (--exclude-ref-prefix subtracts from --all-refs):

For a pure batching test I'd use Option A so you keep the full 55,008 refs. Check how many you'd lose with B first:

Worth noting for real-world use: mirroring Entire refs/entire/checkpoints/* to GitHub is probably undesirable anyway (it exposes session snapshots), so excluding that namespace may be the right default for GitHub targets regardless of this test.

One product question this surfaces: replicate's contract is target==source, so a per-ref push-protection rejection fails the whole run. If you ever want git-sync to skip refs a target rejects (push-protection, protected branches) and continue, that'd be a --best-effort-style mode — not in scope here, but tell me if it's worth a separate issue.

Done. Both changes are implemented, tested, committed, and pushed to PR #94.

1. --target-max-ref-updates flag

Plumbed a per-target batch size cleanly (no global mutable state):

• Pusher.MaxRefUpdates field → package functions take a maxRefUpdates int param → effectiveMaxRefUpdates resolves 0 → env-or-default.
• Wired flag → unstable.Options.TargetMaxRefUpdates → syncer.Config → Pusher, on replicate, sync, plan, bootstrap.
• Precedence: flag > GITSYNC_MAX_REF_UPDATES_PER_PUSH env > default (5,000).

Usage for entire-server targets:

2. Quieted ref-only batch progress

Ref-only follow-up batches now push with progress suppressed (no more bare target: spam). When --verbose, each batch logs one line instead:

Single-batch pushes stay silent (no behavior change for normal pushes), and the first pack batch still shows its real "Resolving deltas" progress.

Verification

• New tests: TestEffectiveMaxRefUpdates, TestPushCommandsVerboseLogsBatches, TestPushPackUsesDefaultLimitWhenZero, plus the existing batching tests refactored to drive an explicit small limit (faster, and they now exercise the param).
• go build, go vet, full suite (./...) all pass.
• Flag appears in --help.

For your 10k retest: that's the value GitHub rejects, so to find/confirm the ceiling use --target-max-ref-updates 10000 (expect 500) and 5000 (works) — same as the env var, now as a first-class flag. Reset the target with gh repo delete/create between runs as discussed.

Same root cause as before — GitHub secret-scanning push protection (GH013) rejecting the refs/entire/checkpoints/* refs. It's on the replicate relay path now (rather than bootstrap) only because the target is already partially populated from your earlier 5k run, so it took the incremental path. Still not a git-sync bug — and replicate's contract is strict (target == source), so one rejected ref fails the whole run.

Three ways forward:

A. Disable push protection on the target — keeps all 55k refs, clean replicate (best for a full-fidelity batching test):

B. Exclude the checkpoint namespace (drops those refs from the mirror):

C. Use sync instead of replicate — sync --all-refs runs in best-effort mode, which downgrades per-ref rejections (like push protection) to warnings and continues, mirroring everything GitHub accepts and skipping what it blocks:

(replicate deliberately omits best-effort; sync includes it with --all-refs.)

For confirming the 10k-vs-5k batching ceiling specifically, I'd do A (so push protection isn't a confounder) on a freshly recreated repo:

then run the sync. Note: a brand-new private repo may have push protection on by GitHub/org default, which is why the explicit disable is worth doing up front.

One thing worth deciding for the product (not this PR): should mirroring to GitHub default to excluding refs/entire/*? Those are Entire's internal checkpoint/session refs — pushing them to an external GitHub remote both trips secret scanning and leaks session snapshots. If the mirror-pipeline targets GitHub, baking in that exclusion (or a "GitHub target ⇒ skip internal namespaces" rule) would avoid this class of failure entirely. Want me to open a separate issue for that?

The most likely culprit is the heredoc: with <<'JSON' the closing JSON must be at the start of the line with no indentation — in your paste it's indented, so the body never terminated correctly. Avoid the heredoc entirely and pipe the JSON in:

Verify it took:

If that still errors, two things to check:

• Wrong owner: is it really Soph (capital S)? Confirm with gh repo view Soph/large-ref-test. The path is case-sensitive in the API.
• Permissions / GHAS: on a private repo, secret-scanning push protection is governed by GitHub Advanced Security / org policy and may not be repo-toggleable — you'd get a 403/422. In that case disabling won't work and you should fall back to excluding the refs.

UI fallback: repo Settings → Code security → Push protection → Disable.

If disabling is blocked, just sidestep it — these are the two that don't depend on push-protection settings:

Tell me the exact error/output from the gh api call if it still fails and I'll pin it down — but for your 10k-vs-5k batching test, Option B or C gets you unblocked right now without fighting GitHub's security settings.

Fixed and pushed. The lint failure was a gofmt issue: adding the TargetMaxRefUpdates field changed the longest-key width in two structs, so gofmt wanted the surrounding fields realigned (internal/syncer/syncer.go and unstable/client.go). golangci-lint itself was already clean.

Verified the full CI lint set now passes:

• gofmt -l -s . → clean
• golangci-lint run ./... → 0 issues
• go mod tidy → no diff
• build + tests → green

Pushed as commit 85aba74b to PR #94.

Tip to catch this locally before pushing: mise run fmt (auto-formats) or mise run lint (runs the same gofmt/gomod/golangci/shellcheck group CI uses).