Serve singular scalar string eq filters with a GIN @> containment predicate#5138
Serve singular scalar string eq filters with a GIN @> containment predicate#5138habdelra wants to merge 2 commits into
@> containment predicate#5138Conversation
…redicate
A singular-path scalar `eq` (e.g. `customer.id`) compiled to
`search_doc -> 'customer' ->> 'id' = $v` — a JSON path-extraction the GIN
`search_doc` index cannot serve, so Postgres heap-scans every row of the
filtered card type and discards the non-matches. Rewrite it as a containment
predicate `search_doc @> {"customer":{"id":$v}}`, which the existing
`gin (search_doc)` index serves directly.
The rewrite is modeled as a two-pass concern, parallel to the existing
table-valued (`json_tree`) machinery:
- Pass 1 (schema-aware): a `JsonContainsQuery` node, routed by `fieldArity` on
cardinality only. The singular branch resolves it; the resolver emits a
`JsonContains` node only when the leaf is a string-valued, non-numeric field,
otherwise it falls back to today's `->>` extraction.
- Pass 2 (dialect): `expressionToSql` renders `JsonContains` per adapter —
Postgres `search_doc @> $n::jsonb`, SQLite `search_doc -> 'a' ->> 'b' = $n`.
Scoped so the rewrite is exactly equivalent to extraction-equality:
- singular paths only — a plural segment anywhere keeps the `json_tree` +
`fullkey LIKE` machinery (containment loses per-element positional binding);
- string, non-numeric leaves only — numeric leaves keep their `::numeric` cast;
- positive polarity only — under `not`, `->>` yields NULL on an absent path
while `@>` yields FALSE, and `NOT NULL` vs `NOT FALSE` diverge, so a
polarity flag is threaded through `filterCondition` and negated `eq` keeps
extraction.
SQLite output is byte-identical to before. Adds an `eq-containment`
Postgres integration test covering each routing path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR optimizes realm-server search for singular-path scalar eq filters by rewriting eligible predicates into Postgres JSONB containment (search_doc @> ...::jsonb) so the existing GIN index on search_doc can be used, while preserving existing semantics for cases where containment is not equivalent (e.g., under not, numeric leaves, plural paths, and SQLite).
Changes:
- Thread filter polarity through filter compilation so the containment rewrite only applies at positive polarity.
- Add a two-pass
JsonContainsQuery→JsonContainsexpression node that resolves schema/serializer details before emitting adapter-specific SQL (@>on Postgres,->/->>extraction on SQLite). - Add a Postgres integration test that asserts both result sets and that the expected SQL predicate form was emitted.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| packages/runtime-common/index-query-engine.ts | Adds filter polarity tracking and a schema-aware containment rewrite for eligible singular-path string eq filters. |
| packages/runtime-common/expression.ts | Introduces JsonContainsQuery/JsonContains nodes and renders them as @> (pg) or extraction equality (sqlite). |
| packages/realm-server/tests/index.ts | Registers the new Postgres integration test in the realm-server test suite. |
| packages/realm-server/tests/eq-containment-integration-test.ts | Adds coverage verifying containment vs extraction behavior across string/numeric/boolean/plural/negated cases and checks emitted SQL. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Host Test Results 1 files 1 suites 1h 50m 1s ⏱️ Results for commit fa5911b. Realm Server Test Results 1 files ±0 1 suites ±0 8m 49s ⏱️ +14s Results for commit fa5911b. ± Comparison against earlier commit 025a5c8. |
…query - Brand code-ref module URLs as RealmResourceIdentifier (via a `ref` helper) and match the DefinitionLookup.lookupDefinition signature (ResolvedCodeRef -> Promise<Definition>), so glint `lint:types` passes. - `lastFilterSql()` now excludes the per-test seeding INSERTs so the `@>`-present / `@>`-absent assertions reflect only the search query. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
What
On the realm-server search path, a singular-path scalar
eqfilter — e.g.eq: { 'customer.id': X }from a query-backedlinksToManygetter — compiled to:That
->>path-extraction cannot use the existing GINsearch_docindex, so Postgres narrows to the card type and then heap-scans every row of that type, discarding the non-matches. This PR rewrites such predicates as a containment:which the existing
gin (search_doc)index serves directly (bitmap index scan instead of a heap scan). On staging this was a ~33× win on the measured filter (200 ms → 6 ms, cold) for the same rows.How
The rewrite is a two-pass concern, modeled parallel to the existing table-valued (
json_tree) machinery:JsonContainsQuerynode, emitted byfieldEqFilter.fieldAritystill routes on cardinality only (singular → this node, plural → the existingjson_tree+fullkey LIKEpath). The resolver emits aJsonContainsnode only when the leaf is a string-valued, non-numeric field; otherwise it falls back to today's->>extraction.expressionToSqlrendersJsonContainsper adapter — Postgressearch_doc @> $n::jsonb(nested object from the path segments), SQLitesearch_doc -> 'a' ->> 'b' = $n. SQLite has no@>, so it keeps the extraction form.Why the scope is exactly these three gates
The rewrite is applied only where
@>containment is provably equivalent to the->>extraction equality it replaces:json_treemachinery —@>'s array containment is order-insensitive and "discard non-matching elements", so it loses the per-element positional binding thatfullkey LIKE '$.…[%]…'enforces.number/big-integer) keep their::numericcast — JSON numeric normalization (5vs5.0) differs from text equality. Booleans never reach the branch (a boolean value isn't a string).->>yields SQLNULLwhile@>yieldsFALSE; identical in a positive filter, but undernotthey diverge (NOT NULLdrops the row,NOT FALSEkeeps it). A polarity flag is threaded throughfilterCondition(flipped at eachnot), and negatedeqkeeps extraction.@>is also anchored from the document root (object containment is key-matched recursively from the top), so{"customer":{"id":X}}matches exactlysearch_doc.customer.id == Xand never a strayidelsewhere — equivalent to the-> … ->>navigation, at any nesting depth.Behavior / compatibility
in,contains,matches, and negated filters are unchanged.Tests
Adds
eq-containment-integration-test.ts, a Postgres integration test that asserts both the result set and which SQL form was emitted (by capturing executed SQL), covering: top-level string eq,linksTo .id, nested contains-composite, numeric, boolean, plural-leaf, interior-plural (contacts.email→json_tree), negated (preserving NULL-on-absent-path row exclusion), and double-negation (back to@>).