Multi-cell Fan-out and Cell Routing Infrastructure

All three research agents are running — one mapping the CLI's existing cell/jurisdiction plumbing (experts + #1592), one dissecting #1616's fan-out implementation, and one reading the backend cell model in entiredb/entire-api plus the BFF fan-out pattern. I'll synthesize a recommendation on whether to build the shared foundation first once they report back.

The #1616 analysis is in — key finding: its multi-cell fan-out logic (group-by-cell, cluster catalog resolution, per-cell client construction, partial-failure handling) is all inlined in search_cmd.go and partially duplicates what the experts command already built, while skipping the placement/active-mirror filtering experts does. Still waiting on the other two agents (CLI cell-infra map and the entiredb/entire-api backend model) before I give you a recommendation.

Second report is in. The CLI map confirms the picture: today there is exactly one cell per request everywhere — experts resolves a single repo → single cell, #1592 adds home-cell routing with data-API fallback, and no data-plane fan-out exists anywhere. It also surfaced a structural wart (the auth package hand-parses /api/v1/clusters because of an auth↔coreapi import cycle) and three duplicated repo-ID resolvers. One agent left: the entiredb/entire-api backend model, which should tell us what invariants a shared fan-out layer can rely on (repo↔cell placement, token audience scope, and how the BFF does its fan-out). I'll write up the recommendation when it lands.

All three investigations are done, and I verified the one suspicious join myself. Here's the synthesis.

TLDR

Yes — I'd build the foundation first, but the "foundation" is a modest extraction, not a framework. #1592 is already part of that foundation and should land first. #1616 should then be reworked on top of a small shared cell-routing package before merge: it currently inlines ~300 lines of generic multi-cell orchestration under codeSearch-prefixed names, mints tokens wastefully (once per cell instead of once per jurisdiction), and joins the repo index to the cluster catalog on a fragile key. Since it's gated behind ENTIRE_CODE_SEARCH=1 and has had no human review yet, now is the cheapest moment to fix the shape.

Why the timing question is real: three routing shapes are converging

The backend model (entiredb + entire-api) defines exactly three request shapes, and the BFF (entire.io/api/src) already has a named helper for each:

1. Repo-scoped → one cell (BFF resolve-cell.ts): a placement lives in exactly one cell; resolve it, mint a jurisdiction token, call that cell. The CLI's version is the experts path (resolveExpertsCellTarget in experts_cell_target.go).
2. User-scoped /me → home cell, never fan-out (BFF resolve-home.ts): activity is forwarded into the user's home region, so /me is a single point-to-point call routed by the home_jurisdiction JWT claim. That's what #1592 adds (entireapi_client.go, ErrNoCellForJurisdiction, the case-folded claim).
3. Repo-set queries → fan out to every hosting cell, merge client-side (BFF code-search.ts): no server-side aggregator exists anywhere — core returns catalogs, cells are strictly local. #1616 is the CLI's first instance of this shape.

The CLI is growing these one command at a time, each with its own seams. #1616 would make it three commands, three sets of plumbing.

Concrete problems with merging #1616 as-is

1. The generic layer is inlined and search-named. groupReposByCell, resolveRepoFilters, the ListRepos→ListClusters→CellTarget resolution, the WaitGroup fan-out with per-cell timeouts, and the partial-failure tracking (FailedJurisdictions) are all in search_cmd.go — none of it is search-specific. The next fan-out command (experts natural-language query is an obvious candidate: cells return 503 for it today precisely because it needs the search index) copies or refactors it. It also duplicates experts' patterns with different choices: a second coreClient interface+seam, different timeout constants, and no placement/active-mirror filtering.

2. Token minting ignores the key backend invariant. Identity tokens are per-jurisdiction, not per-cell — every cell in a jurisdiction accepts the same token. #1616 calls auth.NewEntireAPICellClient once per cell, and each call independently re-runs discovery, login-JWT refresh, and the RFC 8693 exchange. The BFF single-flights and caches one token per jurisdiction. With N cells that's N× redundant auth round trips per search, growing with every future fan-out command.

3. The catalog join is fragile — verified. #1616 groups repos by RepoIndexEntry.Cell (physical cell, e.g. aws-us-east-2) but looks that value up in a map keyed by Cluster.Slug (slugToCluster[cells[i].cell], diff line ~576). The cluster catalog response doesn't expose a cell field at all — RepoIndexEntry carries ClusterSlug for exactly this join, and #1616 doesn't use it. Today slug and cell often coincide, so a mismatch degrades to the jurisdiction fallback (right region, but it triggers a hand-rolled per-cell cluster-catalog re-fetch inside auth) rather than misrouting — but it's a silent behavioral cliff. This is now the third distinct join key against the same catalog: experts joins by PublicUrl host, #1616 by slug-as-cell, the auth fallback by jurisdiction.

4. Existing warts get a third copy instead of a fix. resolveCellAPIBaseURL in auth/cell_data_api.go:508 already hand-parses GET /api/v1/clusters because of the auth↔coreapi import cycle, and repo-ID resolution exists three times (api_cmd.go, experts_cmd.go, #1592's currentRepoID).

What the foundation should be

A small package (say cmd/entire/cli/cellrouting) with three entry points mirroring the BFF's helpers, built from what already exists:

• RepoCell(ctx, fullName, ulid) *auth.CellTarget — lift resolveExpertsCellTarget out of experts naming.
• HomeCell(ctx) — #1592's entireapi_client.go logic, essentially as-is.
• CellsFor(ctx, repoFilter) []CellGroup + FanOut(ctx, groups, timeout, fn) — extracted from #1616's searchAllCells/searchCell, with the join fixed to ClusterSlug↔Slug, and one coreClient seam shared with experts. Merge semantics stay per-command.
• Per-jurisdiction token cache inside or in front of NewEntireAPICellClient: resolve the stored subject once, exchange once per jurisdiction, reuse across cells in a fan-out.

Deferrable: breaking the auth↔coreapi cycle so there's one cluster-catalog parser, and consolidating the three repo-ID resolvers.

Suggested sequencing

1. Merge #1592 now. It's small, it establishes the home-cell shape and ErrNoCellForJurisdiction, and the foundation builds on it rather than replacing it.
2. Extract the routing package (mostly moving code from #1616's diff plus experts), with the per-jurisdiction token cache and the ClusterSlug join fix.
3. Rebase #1616 onto it. The search-specific parts — the codesearch client package, merge/sort/dedup semantics, output rendering, flag handling — are genuinely good and carry over unchanged; its ~940 lines of tests mostly test logic that moves rather than dies.

The counterargument — merge behind the env gate, refactor later — is defensible, but weak here: the PR has had no human review of the orchestration (only two stale Copilot nits), the extraction is cheap while the code is fresh, and gated code has a way of calcifying into the pattern the next command copies.

Now tidy the redundant fold in resolveJurisdiction and its stale doc comment:

Lint clean. Adding a brief architecture note to CLAUDE.md so the new layer is discoverable, then committing:

No response requested.

All caught up — the whole stack is now either merged or waiting only on human review:

Done since we started:

• #1640 (red-main fix) — merged. Main is green again.
• #1592 (activity/recap → home-cell client) — merged, with all five bot findings fixed.
• #1641 (the cell-routing foundation, now based directly on main) — CI fully green, and all five of its bot findings are fixed and replied to: tokenFor now uses per-jurisdiction mint slots so no lock is held across the network exchange, the fan-out catalog lookup got the cellResolveTimeout bound, resolveCellBaseURLs refuses to set a baseURL without a jurisdiction (which would have dialed a foreign cell with a home token), and empty-cell index entries group per-jurisdiction instead of collapsing into one mixed group.
• #1616 coordination — I posted a comment on evis's PR pointing at #1641 and sketching the rebase: the codesearch client, merge/dedup semantics, and rendering carry over unchanged; the inline orchestration collapses onto groupReposByCell/resolveCellBaseURLs/fanOutCells.

One thing worth flagging from #1616's history: it actually had a commit that correctly switched the catalog join to ClusterSlug, which a later commit reverted citing BFF parity — but the BFF templates cell URLs from the cell name rather than doing a catalog lookup, so the parity argument doesn't apply to a Slug-keyed lookup. The comment spells that out so the join doesn't flip back again; #1641 pins it with a slug≠cell test.

The only remaining step is human: #1641 needs a code-owner review from entireio/cli-maintainers (you can't self-approve, and the ruleset blocks admin bypass). Once it lands, #1616 can rebase — I'm happy to do that rebase or review it when the time comes.

Memory updated too. Once evisdren re-approves, #1641 merges and #1616 can rebase onto it — I've already left the rebase guidance on that PR.