Skip to content

fix: guard FTS triggers so only real content changes re-tokenize - #2314

Merged
ascorbic merged 1 commit into
emdash-cms:mainfrom
edrpls:fix/fts-trigger-when-guards
Aug 13, 2026
Merged

fix: guard FTS triggers so only real content changes re-tokenize#2314
ascorbic merged 1 commit into
emdash-cms:mainfrom
edrpls:fix/fts-trigger-when-guards

Conversation

@edrpls

@edrpls edrpls commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Stacked on #2313 (fix/fts-plain-text) — both PRs rewrite the same trigger-generator SQL, so this branch builds on that one to avoid a guaranteed conflict. Review the last commit only; the earlier commits are #2313.

Fixes FTS sync triggers re-tokenizing the whole document on every row UPDATE, even when nothing searchable changed.

The update trigger had no change detection: any UPDATE deleted and re-inserted the document's full index entry. Metadata-only saves — status flips, scheduling, autosave version bumps — and the publish path's rewrite-identical-values UPDATEs each paid full re-tokenization. Measured on the audited production deployment (Macabro festival site, emdash 0.31.1): 49× CPU on metadata-only saves, and re-tokenization was 78–89% of a save's WAL bytes — the dominant replication-volume driver under litestream-style WAL shipping.

The fix adds a WHEN guard to the generated update trigger comparing raw column values with null-safe IS NOT: the trigger fires only when an indexed field, the row's locale, or its trash state (deleted_at) actually changed.

  • deleted_at must stay in the guard or trash/restore stop syncing the index (locked by test).
  • Raw-column comparison stays valid change detection for Portable Text fields whose indexed values are extracted text (fix: index extracted Portable Text prose in FTS, not raw JSON #2313): if the raw JSON didn't change, the extraction didn't either.
  • Value comparison (not UPDATE OF <cols>) is required because the publish path SETs every data column even when values are unchanged — only comparing values suppresses those re-tokenizations (locked by test).
  • No migration of its own: the WHEN guard ships inside fix: index extracted Portable Text prose in FTS, not raw JSON #2313's rebuild migration (064_fts_plain_text), so existing deployments re-tokenize once during that rebuild instead of twice back to back.

Deliberately out of scope (possible follow-up, would need a Discussion per the performance-PR policy): coalescing the publish path's three separate UPDATEs into one. With the guards in place those UPDATEs no longer re-tokenize unless values actually changed, which removes the write amplification this bug is about.

The failing test observes the FTS _data shadow segments byte-for-byte across a metadata-only UPDATE — on the base branch they get rewritten; with the guard they are identical. Companion tests lock the positive paths: searchable-field edits still re-index, publish-shaped identical-value rewrites don't, and trash/restore still add/remove the row.

Found during a measured database audit of a production deployment.

Type of change

  • Bug fix
  • Feature (requires maintainer-approved Discussion)
  • Refactor (no behavior change)
  • Translation
  • Documentation
  • Performance improvement
  • Tests
  • Chore (dependencies, CI, tooling)

Checklist

  • I have read CONTRIBUTING.md
  • pnpm typecheck passes
  • pnpm lint passes
  • pnpm test passes (or targeted tests for my change)
  • pnpm format has been run
  • I have added/updated tests for my changes (if applicable)
  • User-visible strings in the admin UI are wrapped for translation (if applicable). Do not include messages.po changes except in translation PRs — a workflow extracts catalogs on merge to main. — n/a: no admin UI strings changed
  • I have added a changeset (if this PR changes a published package)
  • New features link to an approved Discussion — n/a: bug fix

AI-generated code disclosure

  • This PR includes AI-generated code — model/tool: Claude Fable 5 (Claude Code)

Screenshots / test output

Failing first (on the base branch, before the guard):

× does not re-tokenize on a metadata-only update
  → expected FTS _data segments to be unchanged, got rewritten segments
× does not re-tokenize when the publish path rewrites identical data values

After the fix — write-amplification suite (the publish case drives repo.publish() for real and fails without the guard) and the full search/migration suites:

Tests  124 passed   (tests/integration/search/, all migration suites)
Full packages/core suite: 5081 passed — the single virtual-modules.test.ts failure is
pre-existing on a clean main checkout in this environment (macOS temp-dir realpath).

🤖 Generated with Claude Code

@changeset-bot

changeset-bot Bot commented Jul 31, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: bb79b51

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 17 packages
Name Type
emdash Patch
@emdash-cms/cloudflare Patch
@emdash-cms/sandbox-workerd Patch
@emdash-cms/plugin-mcp-smoke Patch
@emdash-cms/fixture-perf-site Patch
@emdash-cms/perf-demo-site Patch
@emdash-cms/cache-demo-site Patch
@emdash-cms/do-demo-site Patch
@emdash-cms/do-solo-demo-site Patch
@emdash-cms/admin Patch
@emdash-cms/auth Patch
@emdash-cms/blocks Patch
@emdash-cms/gutenberg-to-portable-text Patch
@emdash-cms/x402 Patch
create-emdash Patch
@emdash-cms/auth-atproto Patch
@emdash-cms/plugin-embeds Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions

Copy link
Copy Markdown
Contributor

Scope check

This PR changes 1,267 lines across 12 files. Large PRs are harder to review and more likely to be closed without review.

If this scope is intentional, no action needed. A maintainer will review it. If not, please consider splitting this into smaller PRs.

See CONTRIBUTING.md for contribution guidelines.

@pkg-pr-new

pkg-pr-new Bot commented Jul 31, 2026

Copy link
Copy Markdown

Open in StackBlitz

@emdash-cms/admin

npm i https://pkg.pr.new/@emdash-cms/admin@2314

@emdash-cms/auth

npm i https://pkg.pr.new/@emdash-cms/auth@2314

@emdash-cms/auth-atproto

npm i https://pkg.pr.new/@emdash-cms/auth-atproto@2314

@emdash-cms/blocks

npm i https://pkg.pr.new/@emdash-cms/blocks@2314

@emdash-cms/cloudflare

npm i https://pkg.pr.new/@emdash-cms/cloudflare@2314

@emdash-cms/contentful-to-portable-text

npm i https://pkg.pr.new/@emdash-cms/contentful-to-portable-text@2314

emdash

npm i https://pkg.pr.new/emdash@2314

create-emdash

npm i https://pkg.pr.new/create-emdash@2314

@emdash-cms/gutenberg-to-portable-text

npm i https://pkg.pr.new/@emdash-cms/gutenberg-to-portable-text@2314

@emdash-cms/plugin-cli

npm i https://pkg.pr.new/@emdash-cms/plugin-cli@2314

@emdash-cms/plugin-types

npm i https://pkg.pr.new/@emdash-cms/plugin-types@2314

@emdash-cms/registry-client

npm i https://pkg.pr.new/@emdash-cms/registry-client@2314

@emdash-cms/registry-lexicons

npm i https://pkg.pr.new/@emdash-cms/registry-lexicons@2314

@emdash-cms/registry-verification

npm i https://pkg.pr.new/@emdash-cms/registry-verification@2314

@emdash-cms/sandbox-workerd

npm i https://pkg.pr.new/@emdash-cms/sandbox-workerd@2314

@emdash-cms/x402

npm i https://pkg.pr.new/@emdash-cms/x402@2314

@emdash-cms/plugin-ai-moderation

npm i https://pkg.pr.new/@emdash-cms/plugin-ai-moderation@2314

@emdash-cms/plugin-atproto

npm i https://pkg.pr.new/@emdash-cms/plugin-atproto@2314

@emdash-cms/plugin-audit-log

npm i https://pkg.pr.new/@emdash-cms/plugin-audit-log@2314

@emdash-cms/plugin-color

npm i https://pkg.pr.new/@emdash-cms/plugin-color@2314

@emdash-cms/plugin-embeds

npm i https://pkg.pr.new/@emdash-cms/plugin-embeds@2314

@emdash-cms/plugin-field-kit

npm i https://pkg.pr.new/@emdash-cms/plugin-field-kit@2314

@emdash-cms/plugin-forms

npm i https://pkg.pr.new/@emdash-cms/plugin-forms@2314

@emdash-cms/plugin-webhook-notifier

npm i https://pkg.pr.new/@emdash-cms/plugin-webhook-notifier@2314

commit: bb79b51

@github-actions

github-actions Bot commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Overlapping PRs

This PR modifies files that are also changed by other open PRs:

This may cause merge conflicts or duplicated work. A maintainer will coordinate.

@edrpls
edrpls force-pushed the fix/fts-trigger-when-guards branch from aa3f860 to ee073a7 Compare August 8, 2026 14:15
@edrpls
edrpls marked this pull request as ready for review August 8, 2026 14:17
@github-actions github-actions Bot added the review/needs-review No maintainer or bot review yet label Aug 8, 2026
@edrpls
edrpls force-pushed the fix/fts-trigger-when-guards branch from ee073a7 to 6c2f2f2 Compare August 9, 2026 07:57
@edrpls
edrpls force-pushed the fix/fts-trigger-when-guards branch 9 times, most recently from ea3b387 to 81a42c9 Compare August 11, 2026 18:20
@ascorbic

Copy link
Copy Markdown
Collaborator

@edrpls Thanks! I've merged 2313, so this can be rebased.

The FTS update trigger fired on ANY row UPDATE, deleting and re-inserting
the document's full index entry even when no searchable column changed.
Metadata-only saves — status flips, scheduling, autosave version bumps —
and the publish path's rewrite-identical-values UPDATEs each paid full
re-tokenization: measured 49x CPU on metadata-only saves and 78-89% of a
save's WAL bytes on an audited production deployment.

Add a WHEN guard comparing raw column values with null-safe IS NOT: the
trigger fires only when an indexed value, the row's locale, or its trash
state actually changed. deleted_at stays in the guard so trash/restore
keep syncing the index. Raw-column comparison remains valid change
detection for Portable Text fields whose indexed values are extracted
text. Existing deployments pick up the guarded triggers through the FTS
rebuild migration on the base branch, so upgrades re-tokenize the corpus
once, not twice.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0175x2Nu2m7LZUznaEQUAVQd
@edrpls

edrpls commented Aug 12, 2026

Copy link
Copy Markdown
Contributor Author

The red Smoke check is the known Google Fonts 404 roulette (details on #2313) — this occurrence was Inter in templates/blog-cloudflare (run). Retriggered; everything code-related is green.

@edrpls
edrpls force-pushed the fix/fts-trigger-when-guards branch from b85cff8 to bb79b51 Compare August 12, 2026 14:52

@ascorbic ascorbic left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@ascorbic
ascorbic merged commit 3d89f87 into emdash-cms:main Aug 13, 2026
46 checks passed
@emdashbot emdashbot Bot mentioned this pull request Aug 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants