✓ Fetching regions

Claude Code·Fable 5·nodo·1mo ago·16min·1 Checkpoint·2 file changes·+2/-2·28.4K tokens

✓ Fetching regions entireio/entwine @ aws-us-east-2.entire.io ✗ error entireio/entwine @ aws-ap-southeast-2.entire.io ✓ ready entireio/entwine @ aws-eu-central-1.entire.io ✓ ready

REPO REGION STATUS CLONE URL entireio/entwine aws-us-east-2 (us) error entire://aws-us-east-2.entire.io/gh/entireio/entwine entireio/entwine aws-ap-southeast-2 (au) ready entire://aws-ap-southeast-2.entire.io/gh/entireio/entwine entireio/entwine aws-eu-central-1 (eu) ready entire://aws-eu-central-1.entire.io/gh/entireio/entwine

Clone them: git clone entire://aws-ap-southeast-2.entire.io/gh/entireio/entwine git clone entire://aws-eu-central-1.entire.io/gh/entireio/entwine entireio/entwine @ aws-us-east-2 (us): poll mirror status: decode response: error: code 404: {Schema:{Value:{Scheme:https Opaque: User: Host:us.auth.entire.io Path:/api/v1/schemas/ErrorModel.json Fragment: RawQuery: RawPath: RawFragment: ForceQuery:false OmitHost:false} Set:true} Detail:{Value:mirror not found Set:true} Errors:[] Instance:{Value:{Scheme: Opaque: User: Host: Path: Fragment: RawQuery: RawPath: RawFragment: ForceQuery:false OmitHost:false} Set:false} Status:{Value:404 Set:true} Title:{Value:Not Found Set:true} Type:{Value:{Scheme:about Opaque:blank User: Host: Path: Fragment: RawQuery: RawPath: RawFragment: ForceQuery:false OmitHost:false} Set:true} AdditionalProps:map[]} what is that about?

1mo ago·2m·4 calls

It's two separate things bundled into one scary-looking message — a real (but benign) state fact, and a cosmetic formatting bug in the CLI.

What actually happened

The CLI fans out across all three regions and polls each one's mirror status for entireio/entwine:

  • ap-southeast-2 (au) → ready
  • eu-central-1 (eu) → ready
  • us-east-2 (us) → the status poll came back HTTP 404 with Detail: "mirror not found"

So the meaning is simple: there is no mirror of this repo in the us-east-2 region. The other two regions have a ready mirror; us-east-2 has no mirror record at all, so asking for its status legitimately 404s. That's also why the CLI only offers you clone URLs for the au and eu regions at the bottom — those are the two that exist and are ready.

Whether that's expected depends on intent: either us-east-2 was never provisioned for this repo, or a mirror there was removed / failed to backfill. It's not corruption — it's an absence.

Why it looks so ugly

The second half of the line:

That whole {Schema:{Value:...} ...} blob is a raw Go %+v-style dump of the parsed RFC-7807 problem-details struct (the ErrorModel.json from us.auth.entire.io). The interesting content is just:

  • Status: 404
  • Title: Not Found
  • Detail: mirror not found

The CLI is decoding the problem+json body correctly but then printing the entire struct instead of formatting the Title/Detail fields into a clean message like us-east-2: mirror not found (404). The decode response: wrapper is also misleading — decoding succeeded; it's the underlying response that was a 404. So the noise is a presentation bug in the CLI's error path, not a decode failure.

Notes on what I could and couldn't verify

  • The error strings (poll mirror status, mirror not found) are not in this repo — this checkout is the entwine library (connect/drain/shutdown). The message is emitted by the separate entire CLI, so I couldn't read the exact printing code to confirm the %+v diagnosis from source; it's inferred from the output shape.
  • Your CLI is pointed at production (*.entire.io). My mirror-status tooling here is staging (royalcanin.partial.to / eukanuba.partial.to), so I can't authoritatively confirm the production us-east-2 state from here. For the record, staging currently has no entwine mirror in any region — only entireio/cli and entireio/cli-checkpoints — but that's a different environment and shouldn't be read as the production truth.

Bottom line: functionally, just clone from au or eu — us-east-2 has no mirror for this repo. If you expected one there, that region needs the mirror (re)created. If you want, I can dig into the CLI repo to pin down the exact spot that dumps the struct and clean up the error formatting.

check in the cli repo

1mo ago·3m·14 calls

Confirmed it in the CLI source. There are genuinely two separate things here, and I can now point at the exact lines.

What produced the message

You ran the entire repo mirror create wizard (the "✓ Fetching regions" spinner + Docker-style per-line status + REPO/REGION/STATUS/CLONE URL table are all from repo_mirror_create_wizard.go), selecting entwine into all three regions. For each region it does create-then-poll:

  1. createAndAwaitMirror calls CreateMirror, which succeeded in us-east-2 — that's why its row still shows a clone URL (res.cloneURL is set from the created mirror at repo_mirror_create_wizard.go:595).
  2. It then polls GetMirror(mirrorId) in a loop (awaitMirrorReady, repo_mirror_probe.go:109). That poll came back HTTP 404 "mirror not found" every tick. After maxConsecutivePollErrors (5) consecutive failures it gives up and returns poll mirror status: <err> (repo_mirror_probe.go:130-131).

So the real, functional problem: in us-east-2, CreateMirror acknowledges a mirror that the status endpoint then can't find. The other two regions poll fine and go ready. Note the very first output block in your paste also showed us-east-2 in this exact 404 state — so this isn't a transient blip during one run; that region's mirror record for entwine is persistently missing/broken while create keeps handing back a URL. That's a backend/cluster consistency issue in us-east-2, not a CLI bug — the CLI is reporting it accurately.

Why it printed as a raw struct

That's the CLI bug. The chain is poll mirror status: → ogen's decode response: wrap → ErrorModelStatusCode.Error(), and that Error() is literally:

%+v on the ErrorModel struct dumps every optional field with its {Value:… Set:true} wrapper and the parsed url.URLs — exactly the blob you saw.

The CLI already has the fix for this: coreapi.APIError (client.go:215) pulls Detail out of that struct — it would return just "mirror not found" — and renderCoreError (corecmd.go:454) wraps it. The problem is the wizard's poll-failure branch skips it:

  • createOneMirror's nonTerminal() sets res.err = err raw (repo_mirror_create_wizard.go:618-624)…
  • …whereas the create-failure branch just above it does res.err = renderCoreError(err) (:591).

Then reportMirrorResults prints r.err with %v (:791), so the raw dump leaks. The one-shot path has the same leak (reportOneShotMirror returns the bare err at repo_mirror.go:339 / :341).

The fix

One-line change in nonTerminal() — route the poll error through renderCoreError like the sibling branch does:

renderCoreError/APIError use errors.As, which walks through the poll mirror status: %w wrapping, so it still extracts Detail. After this, the line collapses to:

Bottom line: functionally, us-east-2's mirror for entwine is broken on the server side (create returns a URL, status 404s) — clone from au or eu, and that region needs backend investigation. The scary output itself is a cosmetic CLI bug where the poll-failure path forgets to run the error through renderCoreError.

Want me to make that change in /Users/nodo/work/entire/cli (the nonTerminal() fix, and optionally the matching one-shot path)? It's a different repo than the one we're in, so I held off until you confirm.

yes fix the cli, but also would love to dive into the actual server-side issue after

1mo ago·10m·23 calls

I now have the smoking gun. Here's the complete server-side picture.

Why GetMirror 404s on a mirror that CreateMirror just returned

The two endpoints don't share a lookup. Create writes; the status poll reads through an affiliation gate, and in us-east-2 that read can't see what create wrote.

GetMirror is not a by-ID DB read. core/coreapi/mirrors.go:140 resolves the ID, then calls ListableMirrorRows(accountID) and scans it for the ID; a miss falls through to notFound("mirror not found") (:187). So "mirror not found" literally means "this mirror is not in the caller's visible set as this core sees it right now" — not "no such row."

ListableMirrorRows (core/api/accessors.go:593) is two reads:

  1. authz.LookupResources("repo", "list", "account", accountID) — a SpiceDB reverse lookup of the account's repo#list grants.
  2. Hydrate those IDs from mirror_repos via a follower read, dropping any row with suspended_at set.

The read policy is the key. authz.Store.LookupResources (core/authz/store.go) issues that lookup with staleRead() = minimizeLatency() consistency, and prod builds the store with NewStoreRW(read, write)reads go to a region-local read replica, writes to the primary. The field comment is explicit: "prod leaves it false so reads stay region-local." So when the poll hits the us-east-2 core, LookupResources reads the us-east-2 SpiceDB replica at whatever revision it currently holds, with no wait for freshness.

Create, by contrast, writes the grant to the primary-region SpiceDB writerfinishMirrorPlacementRegisterMirrorAsNativeRepo folds the creator's repo#list seed into one cross-region WriteRelationships (core/regional/repos.go:201). And it re-asserts that seed unconditionally on every call (idempotent TOUCH, :193), so this is not a "retried create loses the grant" bug — a re-create always re-writes it.

Put together: create writes the grant to the primary; the us-east-2 core answers the poll from its local replica under minimize_latency. If that replica is lagging or not replicating, the grant never shows up there, so LookupResources omits the mirror and GetMirror 404s — persistently. eu (your home region, where the write lands) and au (healthy replica) resolve fine. That's exactly the us-only, persistent, "201 + clone URL but 404 on poll" signature you saw — and it would hit any mirror created in us-east-2, not just entwine, and would also drop us-east-2 rows from the region listing.

Ranked hypotheses and how to confirm on prod

  1. us-east-2 SpiceDB read replica stale / not replicating (most consistent with all the evidence). Confirm: from the us-east-2 core, run the same LookupResources("repo","list","account:<yourAccountID>") with fully_consistent consistency and check whether the us entwine mirror ULID appears. If fully-consistent finds it but minimize_latency doesn't → replica lag/outage confirmed. Cross-check SpiceDB replication lag on the read endpoint the us core's NewStoreRW is wired to, and that it points at the right replica (a config divergence would look identical).
  2. The us placement's suspended_at got set after create (async). Suspended rows are filtered out of ListableMirrorRows, so a suspended mirror renders as plain 404 here (never "suspended", because the suspend-aware messaging only fires when the row is present with SuspendedAt). Confirm: SELECT suspended_at FROM mirror_repos for the us placement. (Less likely — a us-only suspension for the same upstream would be odd, and the create response carried Suspended:false or the wizard would have short-circuited to "suspended" instead of polling.)
  3. mirror_repos row genuinely absent in the us read path. Confirm the row exists and isn't behind a broken follower. (Lowest — the table is described as global, and a 201 means the write path completed.)

I can't run these against prod from here — the MCP tools in this session are staging (entire-staging-us/eu, clusters *.partial.to), while your CLI hit prod (*.entire.io). If you point me at prod Honeycomb/Datadog (there are MCP servers wired here), I can pull the us-east-2 GetMirror 404 rate and the SpiceDB replica lag to confirm hypothesis #1 with live data.

A code-level robustness fix worth considering

Independent of the infra trigger: GetMirror's by-ID path makes replica lag indistinguishable from "no such mirror" — a caller polling a mirror they just created can get a hard 404 purely because their grant hasn't replicated to the local replica yet. The codebase already has the pattern to fix this: the deny re-checks (Can, HasPublicAccess) fall back to fullyConsistent() before returning a negative. GetMirror could do the same — on a minimize_latency list-set miss for a specific requested ID, re-check that one (account, repo#list, mirrorID) tuple fully-consistent before 404ing. That turns a replica-lag false-404 into a correct hit, without giving up the minimize_latency policy for the common listing path. It wouldn't fix a truly broken replica (hypothesis #1's root cause), but it would stop the poll from spuriously failing during normal replication windows.

Want me to pull the prod telemetry to confirm the replica-lag hypothesis, sketch that GetMirror fully-consistent-fallback fix in entiredb, or both?