✓ Fetching regions

✓ Fetching regions entireio/entwine @ aws-us-east-2.entire.io ✗ error entireio/entwine @ aws-ap-southeast-2.entire.io ✓ ready entireio/entwine @ aws-eu-central-1.entire.io ✓ ready
REPO REGION STATUS CLONE URL entireio/entwine aws-us-east-2 (us) error entire://aws-us-east-2.entire.io/gh/entireio/entwine entireio/entwine aws-ap-southeast-2 (au) ready entire://aws-ap-southeast-2.entire.io/gh/entireio/entwine entireio/entwine aws-eu-central-1 (eu) ready entire://aws-eu-central-1.entire.io/gh/entireio/entwine
Clone them: git clone entire://aws-ap-southeast-2.entire.io/gh/entireio/entwine git clone entire://aws-eu-central-1.entire.io/gh/entireio/entwine entireio/entwine @ aws-us-east-2 (us): poll mirror status: decode response: error: code 404: {Schema:{Value:{Scheme:https Opaque: User: Host:us.auth.entire.io Path:/api/v1/schemas/ErrorModel.json Fragment: RawQuery: RawPath: RawFragment: ForceQuery:false OmitHost:false} Set:true} Detail:{Value:mirror not found Set:true} Errors:[] Instance:{Value:{Scheme: Opaque: User: Host: Path: Fragment: RawQuery: RawPath: RawFragment: ForceQuery:false OmitHost:false} Set:false} Status:{Value:404 Set:true} Title:{Value:Not Found Set:true} Type:{Value:{Scheme:about Opaque:blank User: Host: Path: Fragment: RawQuery: RawPath: RawFragment: ForceQuery:false OmitHost:false} Set:true} AdditionalProps:map[]} what is that about?
It's two separate things bundled into one scary-looking message — a real (but benign) state fact, and a cosmetic formatting bug in the CLI.
What actually happened
The CLI fans out across all three regions and polls each one's mirror status for entireio/entwine:
- ap-southeast-2 (au) → ready
- eu-central-1 (eu) → ready
- us-east-2 (us) → the status poll came back HTTP 404 with
Detail: "mirror not found"
So the meaning is simple: there is no mirror of this repo in the us-east-2 region. The other two regions have a ready mirror; us-east-2 has no mirror record at all, so asking for its status legitimately 404s. That's also why the CLI only offers you clone URLs for the au and eu regions at the bottom — those are the two that exist and are ready.
Whether that's expected depends on intent: either us-east-2 was never provisioned for this repo, or a mirror there was removed / failed to backfill. It's not corruption — it's an absence.
Why it looks so ugly
The second half of the line:
That whole {Schema:{Value:...} ...} blob is a raw Go %+v-style dump of the parsed RFC-7807 problem-details struct (the ErrorModel.json from us.auth.entire.io). The interesting content is just:
Status: 404Title: Not FoundDetail: mirror not found
The CLI is decoding the problem+json body correctly but then printing the entire struct instead of formatting the Title/Detail fields into a clean message like us-east-2: mirror not found (404). The decode response: wrapper is also misleading — decoding succeeded; it's the underlying response that was a 404. So the noise is a presentation bug in the CLI's error path, not a decode failure.
Notes on what I could and couldn't verify
- The error strings (
poll mirror status,mirror not found) are not in this repo — this checkout is theentwinelibrary (connect/drain/shutdown). The message is emitted by the separateentireCLI, so I couldn't read the exact printing code to confirm the%+vdiagnosis from source; it's inferred from the output shape. - Your CLI is pointed at production (
*.entire.io). My mirror-status tooling here is staging (royalcanin.partial.to/eukanuba.partial.to), so I can't authoritatively confirm the production us-east-2 state from here. For the record, staging currently has noentwinemirror in any region — onlyentireio/cliandentireio/cli-checkpoints— but that's a different environment and shouldn't be read as the production truth.
Bottom line: functionally, just clone from au or eu — us-east-2 has no mirror for this repo. If you expected one there, that region needs the mirror (re)created. If you want, I can dig into the CLI repo to pin down the exact spot that dumps the struct and clean up the error formatting.
Confirmed it in the CLI source. There are genuinely two separate things here, and I can now point at the exact lines.
What produced the message
You ran the entire repo mirror create wizard (the "✓ Fetching regions" spinner + Docker-style per-line status + REPO/REGION/STATUS/CLONE URL table are all from repo_mirror_create_wizard.go), selecting entwine into all three regions. For each region it does create-then-poll:
createAndAwaitMirrorcallsCreateMirror, which succeeded in us-east-2 — that's why its row still shows a clone URL (res.cloneURLis set from the created mirror atrepo_mirror_create_wizard.go:595).- It then polls
GetMirror(mirrorId)in a loop (awaitMirrorReady,repo_mirror_probe.go:109). That poll came back HTTP 404 "mirror not found" every tick. AftermaxConsecutivePollErrors(5) consecutive failures it gives up and returnspoll mirror status: <err>(repo_mirror_probe.go:130-131).
So the real, functional problem: in us-east-2, CreateMirror acknowledges a mirror that the status endpoint then can't find. The other two regions poll fine and go ready. Note the very first output block in your paste also showed us-east-2 in this exact 404 state — so this isn't a transient blip during one run; that region's mirror record for entwine is persistently missing/broken while create keeps handing back a URL. That's a backend/cluster consistency issue in us-east-2, not a CLI bug — the CLI is reporting it accurately.
Why it printed as a raw struct
That's the CLI bug. The chain is poll mirror status: → ogen's decode response: wrap → ErrorModelStatusCode.Error(), and that Error() is literally:
%+v on the ErrorModel struct dumps every optional field with its {Value:… Set:true} wrapper and the parsed url.URLs — exactly the blob you saw.
The CLI already has the fix for this: coreapi.APIError (client.go:215) pulls Detail out of that struct — it would return just "mirror not found" — and renderCoreError (corecmd.go:454) wraps it. The problem is the wizard's poll-failure branch skips it:
createOneMirror'snonTerminal()setsres.err = errraw (repo_mirror_create_wizard.go:618-624)…- …whereas the create-failure branch just above it does
res.err = renderCoreError(err)(:591).
Then reportMirrorResults prints r.err with %v (:791), so the raw dump leaks. The one-shot path has the same leak (reportOneShotMirror returns the bare err at repo_mirror.go:339 / :341).
The fix
One-line change in nonTerminal() — route the poll error through renderCoreError like the sibling branch does:
renderCoreError/APIError use errors.As, which walks through the poll mirror status: %w wrapping, so it still extracts Detail. After this, the line collapses to:
Bottom line: functionally, us-east-2's mirror for entwine is broken on the server side (create returns a URL, status 404s) — clone from au or eu, and that region needs backend investigation. The scary output itself is a cosmetic CLI bug where the poll-failure path forgets to run the error through renderCoreError.
Want me to make that change in /Users/nodo/work/entire/cli (the nonTerminal() fix, and optionally the matching one-shot path)? It's a different repo than the one we're in, so I held off until you confirm.
yes fix the cli, but also would love to dive into the actual server-side issue after
I now have the smoking gun. Here's the complete server-side picture.
Why GetMirror 404s on a mirror that CreateMirror just returned
The two endpoints don't share a lookup. Create writes; the status poll reads through an affiliation gate, and in us-east-2 that read can't see what create wrote.
GetMirror is not a by-ID DB read. core/coreapi/mirrors.go:140 resolves the ID, then calls ListableMirrorRows(accountID) and scans it for the ID; a miss falls through to notFound("mirror not found") (:187). So "mirror not found" literally means "this mirror is not in the caller's visible set as this core sees it right now" — not "no such row."
ListableMirrorRows (core/api/accessors.go:593) is two reads:
authz.LookupResources("repo", "list", "account", accountID)— a SpiceDB reverse lookup of the account'srepo#listgrants.- Hydrate those IDs from
mirror_reposvia a follower read, dropping any row withsuspended_atset.
The read policy is the key. authz.Store.LookupResources (core/authz/store.go) issues that lookup with staleRead() = minimizeLatency() consistency, and prod builds the store with NewStoreRW(read, write) — reads go to a region-local read replica, writes to the primary. The field comment is explicit: "prod leaves it false so reads stay region-local." So when the poll hits the us-east-2 core, LookupResources reads the us-east-2 SpiceDB replica at whatever revision it currently holds, with no wait for freshness.
Create, by contrast, writes the grant to the primary-region SpiceDB writer — finishMirrorPlacement → RegisterMirrorAsNativeRepo folds the creator's repo#list seed into one cross-region WriteRelationships (core/regional/repos.go:201). And it re-asserts that seed unconditionally on every call (idempotent TOUCH, :193), so this is not a "retried create loses the grant" bug — a re-create always re-writes it.
Put together: create writes the grant to the primary; the us-east-2 core answers the poll from its local replica under minimize_latency. If that replica is lagging or not replicating, the grant never shows up there, so LookupResources omits the mirror and GetMirror 404s — persistently. eu (your home region, where the write lands) and au (healthy replica) resolve fine. That's exactly the us-only, persistent, "201 + clone URL but 404 on poll" signature you saw — and it would hit any mirror created in us-east-2, not just entwine, and would also drop us-east-2 rows from the region listing.
Ranked hypotheses and how to confirm on prod
- us-east-2 SpiceDB read replica stale / not replicating (most consistent with all the evidence). Confirm: from the us-east-2 core, run the same
LookupResources("repo","list","account:<yourAccountID>")with fully_consistent consistency and check whether the usentwinemirror ULID appears. If fully-consistent finds it butminimize_latencydoesn't → replica lag/outage confirmed. Cross-check SpiceDB replication lag on thereadendpoint the us core'sNewStoreRWis wired to, and that it points at the right replica (a config divergence would look identical). - The us placement's
suspended_atgot set after create (async). Suspended rows are filtered out ofListableMirrorRows, so a suspended mirror renders as plain 404 here (never "suspended", because the suspend-aware messaging only fires when the row is present withSuspendedAt). Confirm:SELECT suspended_at FROM mirror_reposfor the us placement. (Less likely — a us-only suspension for the same upstream would be odd, and the create response carriedSuspended:falseor the wizard would have short-circuited to "suspended" instead of polling.) mirror_reposrow genuinely absent in the us read path. Confirm the row exists and isn't behind a broken follower. (Lowest — the table is described as global, and a 201 means the write path completed.)
I can't run these against prod from here — the MCP tools in this session are staging (entire-staging-us/eu, clusters *.partial.to), while your CLI hit prod (*.entire.io). If you point me at prod Honeycomb/Datadog (there are MCP servers wired here), I can pull the us-east-2 GetMirror 404 rate and the SpiceDB replica lag to confirm hypothesis #1 with live data.
A code-level robustness fix worth considering
Independent of the infra trigger: GetMirror's by-ID path makes replica lag indistinguishable from "no such mirror" — a caller polling a mirror they just created can get a hard 404 purely because their grant hasn't replicated to the local replica yet. The codebase already has the pattern to fix this: the deny re-checks (Can, HasPublicAccess) fall back to fullyConsistent() before returning a negative. GetMirror could do the same — on a minimize_latency list-set miss for a specific requested ID, re-check that one (account, repo#list, mirrorID) tuple fully-consistent before 404ing. That turns a replica-lag false-404 into a correct hit, without giving up the minimize_latency policy for the common listing path. It wouldn't fix a truly broken replica (hypothesis #1's root cause), but it would stop the poll from spuriously failing during normal replication windows.
Want me to pull the prod telemetry to confirm the replica-lag hypothesis, sketch that GetMirror fully-consistent-fallback fix in entiredb, or both?