feat(search): add POST /v1/search with dirctl search parity

[akijakya]

Closes #1905

Summary

Adds POST /v1/search to the AI Catalog gateway: free text in, relevance-ranked CatalogEntry results out. The acceptance bar is parity with dirctl search "<query>" — the same query against the same node returns the same records in the same order through either path.

Commits, grouped by the risk they carry:

1. refactor(nlsearch) — move the existing fan-out into utils/nlsearch and make ranking deterministic. Touches dirctl search.
2. feat(search) — the endpoint itself. New surface only.
3. fix(search), feat(search), fix(search) — review follow-ups: query length enforcement, projectable-candidate restriction, rejecting unusable queries, and counting the query limit in characters rather than bytes.

Rebased onto main after #2023

POST /v1/extract merged first, and both endpoints live on the same service, so this branch was rebased onto it. Three things came out of that and are worth knowing while reviewing:

• Both RPCs coexist in the proto, and the bindings were regenerated from the merged file rather than hand-resolved — a fresh task api:gen produces no diff.
• fakeExtractor is now shared. It arrived on main with the ExtractTaxonomy tests; the search tests reuse it instead of declaring a second one in the same package.
• WithExtractor's doc names both extractor-backed RPCs, and the //nolint:unused that guarded the ext field while it had no consumer is gone — main removed it when extract became the first one.

Heads-up: this changes dirctl search

The natural-language search algorithm is not new and is not changing. dirctl search "some phrase" already decomposed the phrase into signals, queried each one independently, and ranked the union by how many signals matched — that is what its --help has always described. The first commit moves that code from cli/cmd/search into utils/nlsearch so the gateway can run the same implementation.

One behavior does change for the CLI: tie ordering is now stable.

The scorer accumulated results in goroutine-completion order and then sorted only by hit count, so records with equal scores came back in a different order on every run. Printing results once, nobody notices. For a paginated caller it is a correctness bug — an unstable order lets a record appear on two pages, or on none. Ranking is now a total order: hit count, then summed signal score, then CID.

So the CLI's results are the same set, in the same relevance tiers, with ties no longer shuffling between invocations.

How the search works

The extractor turns one phrase into several signals. "review my python code" might yield two skills, a domain, and a keyword. Each is queried independently and concurrently, and the union is ranked by how many signals matched:

code_review        → {a, b}
static_analysis    → {b}
software_dev       → {b, c}
python (name∪desc) → {c, d}
                     ↓ union + count
b:3   c:2   a:1   d:1

The union matters. ANDing the signals — requiring every guess the extractor made to be simultaneously correct — usually returns nothing; that is why a node holding four cloud-tagged records could answer "cloud computing" with an empty list. Ranking, not exclusion, is what separates a record matching four signals from one matching a single signal.

Cost: N concurrent queries per request rather than one, where N is the signal count (roughly 4–10 at the default two tiers). On the gateway these are in-process rather than gRPC round trips, and each is capped at 500 candidates.

Deliberate choices

• No filter parameter. dirctl search applies no facets to a natural-language query, so accepting them here would break parity. Structured filtering stays on GET /v1/agents?filter=. This also removes the facet-versus-pagination problem the earlier design ran into.
• No displayName fallback on empty extraction. The endpoint was originally specified with a substring fallback; it is dropped. It answers a different question than the one asked and would return hits where dirctl search returns none, breaking the parity this endpoint exists for. [Feature]: Backend POST /v1/search — natural-language search with dirctl parity #1905 has been updated to match.
• An unusable query is rejected, not answered with an empty page. When the extractor derives no signals, INVALID_ARGUMENT is returned with the advice the CLI gives. Zero results cannot distinguish "nothing in the catalog matches" from "the query was not understood", and only the second is fixed by rephrasing.
• Candidates are restricted to catalog-projectable records. GetRecordCIDs searches every record, but GetCatalogEntries only projects records carrying a known catalog module. Without the restriction, unprojectable records occupied page slots and inflated total_count, then vanished at hydration. Exact CID-set parity with the CLI is not achievable here — it returns raw CIDs for any record, this returns CatalogEntry — so parity holds over the projectable universe, made explicit in the query rather than appearing as records quietly disappearing.
• Query length is enforced in the handler, in characters. The service registers no protovalidate interceptor, so the proto's max_len: 1024 was not applied at runtime; enforced in code as ListAgents does with filterMaxLen. Counted with utf8.RuneCountInString, matching how protovalidate reads max_len and how ExtractTaxonomy enforces the same limit — a byte count would have rejected multi-byte queries at roughly a third of the documented length.
• A failing signal degrades recall, not the request. It is logged and the remaining signals still contribute, so one unhealthy query does not turn into a 500.
• total_count is candidates found, bounded by the per-signal fan-out cap — not a whole-index count.
• 503 when no extractor is configured, so callers can distinguish "not set up" from "no matches".

Testing

• 6 unit tests on the shared fan-out: union rather than intersection, hit-count ranking, keyword NAME∪DESCRIPTION dedup (one keyword counts once), per-signal error isolation, limit defaulting and override, and determinism asserted across 25 identical runs
• 11 handler tests: rank order preserved through hydration, pagination disjoint and complete, offset past the end, unusable-query rejection, projectable-candidate restriction, 503 paths, invalid requests including over-length queries, search failure degrading rather than erroring, and a full-length multi-byte query being accepted
• All 6 modules build; task lint 0 issues (Go and Helm); task test 0 failures
• The refactor commit builds and passes tests standalone, so the split is bisectable

Not covered: relevance quality against a seeded catalog. The tests pin the mechanism, not whether the taxonomy matches are good.

Follow-ups

• [Feature]: Backend POST /v1/extract — expose OASF extractor output for tag recommendation #2001 — POST /v1/extract, the endpoint the AI Catalog UI will use for tag recommendation
• [Test]: e2e for POST /v1/search parity and POST /v1/extract #1909 — e2e for both endpoints, including a dirctl search vs POST /v1/search parity assertion
• [Feature]: Helm: optional OASF-SDK extractor server deployment + config #1907 — optional OASF-SDK deployment via Helm

[github-actions]

The latest Buf updates on your PR. Results from workflow Buf CI / verify-proto (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
✅ passed	⏩ skipped	⏩ skipped	✅ passed	Aug 14, 2026, 1:14 PM

[Copilot]

Pull request overview

Adds natural-language catalog search with shared CLI/gateway ranking and pagination.

Changes:

• Centralizes concurrent fan-out and deterministic scoring.
• Adds POST /v1/search and gateway handling.
• Adds unit tests for ranking, failures, and pagination.

Reviewed changes

Copilot reviewed 8 out of 11 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
utils/nlsearch/fanout.go	Implements shared fan-out and ranking.
utils/nlsearch/fanout_test.go	Tests shared search behavior.
server/controller/ai_finder.go	Adds search database and extractor dependencies.
server/controller/ai_finder_test.go	Extends the fake catalog database.
server/controller/ai_finder_search.go	Implements the search endpoint.
server/controller/ai_finder_search_test.go	Tests endpoint behavior.
proto/agntcy/dir/catalog/v1/ai_finder_service.proto	Defines the search API.
cli/cmd/search/nlsearch.go	Migrates CLI search to shared ranking.
api/catalog/v1/ai_finder_service.pb.gw.go	Adds generated HTTP routing.
api/catalog/v1/ai_finder_service.pb.go	Adds generated search messages.
api/catalog/v1/ai_finder_service_grpc.pb.go	Adds generated gRPC bindings.

Files not reviewed (3)

• api/catalog/v1/ai_finder_service.pb.go: Generated file
• api/catalog/v1/ai_finder_service.pb.gw.go: Generated file
• api/catalog/v1/ai_finder_service_grpc.pb.go: Generated file

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[codecov]

Codecov Report

❌ Patch coverage is 58.54701% with 97 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
api/catalog/v1/ai_finder_service.pb.gw.go	0.0%	50 Missing and 1 partial ⚠️
cli/cmd/search/nlsearch.go	0.0%	35 Missing ⚠️
server/controller/ai_finder_search.go	88.8%	5 Missing and 4 partials ⚠️
utils/nlsearch/fanout.go	97.0%	1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!