fix: guard FTS triggers so only real content changes re-tokenize - #2314
Conversation
🦋 Changeset detectedLatest commit: bb79b51 The changes in this PR will be included in the next version bump. This PR includes changesets to release 17 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Scope checkThis PR changes 1,267 lines across 12 files. Large PRs are harder to review and more likely to be closed without review. If this scope is intentional, no action needed. A maintainer will review it. If not, please consider splitting this into smaller PRs. See CONTRIBUTING.md for contribution guidelines. |
@emdash-cms/admin
@emdash-cms/auth
@emdash-cms/auth-atproto
@emdash-cms/blocks
@emdash-cms/cloudflare
@emdash-cms/contentful-to-portable-text
emdash
create-emdash
@emdash-cms/gutenberg-to-portable-text
@emdash-cms/plugin-cli
@emdash-cms/plugin-types
@emdash-cms/registry-client
@emdash-cms/registry-lexicons
@emdash-cms/registry-verification
@emdash-cms/sandbox-workerd
@emdash-cms/x402
@emdash-cms/plugin-ai-moderation
@emdash-cms/plugin-atproto
@emdash-cms/plugin-audit-log
@emdash-cms/plugin-color
@emdash-cms/plugin-embeds
@emdash-cms/plugin-field-kit
@emdash-cms/plugin-forms
@emdash-cms/plugin-webhook-notifier
commit: |
Overlapping PRsThis PR modifies files that are also changed by other open PRs:
This may cause merge conflicts or duplicated work. A maintainer will coordinate. |
aa3f860 to
ee073a7
Compare
ee073a7 to
6c2f2f2
Compare
ea3b387 to
81a42c9
Compare
|
@edrpls Thanks! I've merged 2313, so this can be rebased. |
81a42c9 to
b85cff8
Compare
The FTS update trigger fired on ANY row UPDATE, deleting and re-inserting the document's full index entry even when no searchable column changed. Metadata-only saves — status flips, scheduling, autosave version bumps — and the publish path's rewrite-identical-values UPDATEs each paid full re-tokenization: measured 49x CPU on metadata-only saves and 78-89% of a save's WAL bytes on an audited production deployment. Add a WHEN guard comparing raw column values with null-safe IS NOT: the trigger fires only when an indexed value, the row's locale, or its trash state actually changed. deleted_at stays in the guard so trash/restore keep syncing the index. Raw-column comparison remains valid change detection for Portable Text fields whose indexed values are extracted text. Existing deployments pick up the guarded triggers through the FTS rebuild migration on the base branch, so upgrades re-tokenize the corpus once, not twice. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0175x2Nu2m7LZUznaEQUAVQd
|
The red Smoke check is the known Google Fonts 404 roulette (details on #2313) — this occurrence was Inter in |
b85cff8 to
bb79b51
Compare
What does this PR do?
Fixes FTS sync triggers re-tokenizing the whole document on every row UPDATE, even when nothing searchable changed.
The update trigger had no change detection: any UPDATE deleted and re-inserted the document's full index entry. Metadata-only saves — status flips, scheduling, autosave version bumps — and the publish path's rewrite-identical-values UPDATEs each paid full re-tokenization. Measured on the audited production deployment (Macabro festival site, emdash 0.31.1): 49× CPU on metadata-only saves, and re-tokenization was 78–89% of a save's WAL bytes — the dominant replication-volume driver under litestream-style WAL shipping.
The fix adds a
WHENguard to the generated update trigger comparing raw column values with null-safeIS NOT: the trigger fires only when an indexed field, the row'slocale, or its trash state (deleted_at) actually changed.deleted_atmust stay in the guard or trash/restore stop syncing the index (locked by test).UPDATE OF <cols>) is required because the publish path SETs every data column even when values are unchanged — only comparing values suppresses those re-tokenizations (locked by test).064_fts_plain_text), so existing deployments re-tokenize once during that rebuild instead of twice back to back.Deliberately out of scope (possible follow-up, would need a Discussion per the performance-PR policy): coalescing the publish path's three separate UPDATEs into one. With the guards in place those UPDATEs no longer re-tokenize unless values actually changed, which removes the write amplification this bug is about.
The failing test observes the FTS
_datashadow segments byte-for-byte across a metadata-only UPDATE — on the base branch they get rewritten; with the guard they are identical. Companion tests lock the positive paths: searchable-field edits still re-index, publish-shaped identical-value rewrites don't, and trash/restore still add/remove the row.Found during a measured database audit of a production deployment.
Type of change
Checklist
pnpm typecheckpassespnpm lintpassespnpm testpasses (or targeted tests for my change)pnpm formathas been runmessages.pochanges except in translation PRs — a workflow extracts catalogs on merge tomain. — n/a: no admin UI strings changedAI-generated code disclosure
Screenshots / test output
Failing first (on the base branch, before the guard):
After the fix — write-amplification suite (the publish case drives repo.publish() for real and fails without the guard) and the full search/migration suites:
🤖 Generated with Claude Code