fix: index extracted Portable Text prose in FTS, not raw JSON

[edrpls]

What does this PR do?

Fixes full-text search indexing raw Portable Text JSON instead of prose. On the audited production deployment (Macabro festival site, emdash 0.31.1), structural tokens were 27–29% of the FTS index: searching normal (a PT style value) matched 870/906 posts, _type matched 2,096 documents, and snippets showed JSON fragments.

Why the fix isn't "just call extractPlainText": the FTS sync is done by SQL triggers, which can't call into JS — and the tables were external-content FTS5 (content='ec_<slug>'), which requires the index to exactly mirror the raw column values (snippet() reads them, and the 'delete' command must be fed the inserted values or the index corrupts — the exact corruption class migration 039 exists to fix). So indexing extracted text under external content is structurally impossible without materializing it into real columns on every ec_* table, which would touch every content write path in the codebase.

Instead, this PR rebuilds the FTS tables as self-contained FTS5 whose Portable Text columns hold extracted prose, with the extraction done in SQL so triggers and population share one source of truth:

• json_tree() collects every JSON string under a text, alt, caption, or code key — span text, image alt/caption, and code blocks, the same semantics as extractPlainText in text-extraction.ts. json_valid() guards legacy rows holding a bare string (indexed as-is).
• Self-contained tables retire the external-content 'delete' choreography entirely: removal is a plain DELETE, a harmless no-op for never-indexed (soft-deleted) rows, so the SQLITE_CORRUPT_VTAB corruption class from SQLite Database Corrupt Error on new site #649/migration 039 can no longer occur. INSERT OR REPLACE makes concurrent D1 populates converge (verified: a plain duplicate-rowid insert throws on self-contained FTS5).
• Snippets now come from the stored prose — searching content returns readable text, not {"_type":"block"....

Existing deployments rebuild automatically: migration 064_fts_plain_text drops and recreates every search-enabled collection's FTS table and triggers and repopulates from content, following 039's structure (self-contained lock-step SQL copy, IF NOT EXISTS forms for D1's lockless concurrent migrators, defensive identifier validation). No manual reindex step. The rebuild keeps each collection's configured tokenize from search_config (added on main by #2257) instead of resetting it to the default.

The search query layer is unchanged: it joins ec_* by id for metadata, bm25 weight positions and the snippet column index are preserved (same column order), and Postgres is unaffected (FTS5 is SQLite-gated throughout). Trade-off worth noting for review: a self-contained FTS table stores its own copy of the indexed text (external content stored none), while the inverted index itself shrinks by the removed structural tokens; net size impact on the audited data set is roughly neutral, and the correctness/corruption-immunity gains are structural.

The 039 migration test's fixture was updated to explicitly construct the historical external-content table shape it needs (the current FTSManager now builds self-contained tables, on which the historical "broken" triggers are actually correct) — 039's own frozen migration SQL is untouched.

The write-amplification WHEN guard (#2314) ships inside this PR's rebuild migration, so an upgrade re-tokenizes the corpus once, not twice; #2314 itself is now manager + tests only. The rebuild also honors each collection's configured tokenize (frozen allowlist, no live import) and keeps legacy scalar values raw-indexed (json_type shape gate).

Found during a measured database audit of a production deployment.

Type of change

• Bug fix
• Feature (requires maintainer-approved Discussion)
• Refactor (no behavior change)
• Translation
• Documentation
• Performance improvement
• Tests
• Chore (dependencies, CI, tooling)

Checklist

• I have read CONTRIBUTING.md
• pnpm typecheck passes
• pnpm lint passes
• pnpm test passes (or targeted tests for my change)
• pnpm format has been run
• I have added/updated tests for my changes (if applicable)
• User-visible strings in the admin UI are wrapped for translation (if applicable). Do not include messages.po changes except in translation PRs — a workflow extracts catalogs on merge to main. — n/a: no admin UI strings changed
• I have added a changeset (if this PR changes a published package)
• New features link to an approved Discussion — n/a: bug fix

AI-generated code disclosure

• This PR includes AI-generated code — model/tool: Claude Fable 5 (Claude Code)

Screenshots / test output

Failing first (on main, before the fix):

× does not match Portable Text structural tokens
  → "normal" must not match: expected [ …(1) ] to deeply equal []
× returns prose snippets, not JSON fragments

After the fix — new search-quality suite, migration 064 upgrade-path suite (pre-migration assertion proves the polluted baseline, post-migration assertions prove the cleanup and working triggers), plus all existing search/FTS/registry/migration suites:

Tests  36 passed   (tests/integration/search/, 039 migration suite)
Tests  17 passed   (064 migration suite, migrations integration)
Full packages/core suite: 5074 passed — the single virtual-modules.test.ts failure is
pre-existing on a clean main checkout in this environment (macOS temp-dir realpath).

🤖 Generated with Claude Code

[changeset-bot]

🦋 Changeset detected

Latest commit: cacf517

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 17 packages

Name	Type
emdash	Patch
@emdash-cms/cloudflare	Patch
@emdash-cms/sandbox-workerd	Patch
@emdash-cms/plugin-mcp-smoke	Patch
@emdash-cms/fixture-perf-site	Patch
@emdash-cms/perf-demo-site	Patch
@emdash-cms/cache-demo-site	Patch
@emdash-cms/do-demo-site	Patch
@emdash-cms/do-solo-demo-site	Patch
@emdash-cms/admin	Patch
@emdash-cms/auth	Patch
@emdash-cms/blocks	Patch
@emdash-cms/gutenberg-to-portable-text	Patch
@emdash-cms/x402	Patch
create-emdash	Patch
@emdash-cms/auth-atproto	Patch
@emdash-cms/plugin-embeds	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[github-actions]

Scope check

This PR changes 772 lines across 8 files. Large PRs are harder to review and more likely to be closed without review.

If this scope is intentional, no action needed. A maintainer will review it. If not, please consider splitting this into smaller PRs.

See CONTRIBUTING.md for contribution guidelines.

[pkg-pr-new]

Open in StackBlitz

@emdash-cms/admin

npm i https://pkg.pr.new/@emdash-cms/admin@2313

@emdash-cms/auth

npm i https://pkg.pr.new/@emdash-cms/auth@2313

@emdash-cms/auth-atproto

npm i https://pkg.pr.new/@emdash-cms/auth-atproto@2313

@emdash-cms/blocks

npm i https://pkg.pr.new/@emdash-cms/blocks@2313

@emdash-cms/cloudflare

npm i https://pkg.pr.new/@emdash-cms/cloudflare@2313

@emdash-cms/contentful-to-portable-text

npm i https://pkg.pr.new/@emdash-cms/contentful-to-portable-text@2313

emdash

npm i https://pkg.pr.new/emdash@2313

create-emdash

npm i https://pkg.pr.new/create-emdash@2313

@emdash-cms/gutenberg-to-portable-text

npm i https://pkg.pr.new/@emdash-cms/gutenberg-to-portable-text@2313

@emdash-cms/plugin-cli

npm i https://pkg.pr.new/@emdash-cms/plugin-cli@2313

@emdash-cms/plugin-types

npm i https://pkg.pr.new/@emdash-cms/plugin-types@2313

@emdash-cms/registry-client

npm i https://pkg.pr.new/@emdash-cms/registry-client@2313

@emdash-cms/registry-lexicons

npm i https://pkg.pr.new/@emdash-cms/registry-lexicons@2313

@emdash-cms/registry-verification

npm i https://pkg.pr.new/@emdash-cms/registry-verification@2313

@emdash-cms/sandbox-workerd

npm i https://pkg.pr.new/@emdash-cms/sandbox-workerd@2313

@emdash-cms/x402

npm i https://pkg.pr.new/@emdash-cms/x402@2313

@emdash-cms/plugin-ai-moderation

npm i https://pkg.pr.new/@emdash-cms/plugin-ai-moderation@2313

@emdash-cms/plugin-atproto

npm i https://pkg.pr.new/@emdash-cms/plugin-atproto@2313

@emdash-cms/plugin-audit-log

npm i https://pkg.pr.new/@emdash-cms/plugin-audit-log@2313

@emdash-cms/plugin-color

npm i https://pkg.pr.new/@emdash-cms/plugin-color@2313

@emdash-cms/plugin-embeds

npm i https://pkg.pr.new/@emdash-cms/plugin-embeds@2313

@emdash-cms/plugin-field-kit

npm i https://pkg.pr.new/@emdash-cms/plugin-field-kit@2313

@emdash-cms/plugin-forms

npm i https://pkg.pr.new/@emdash-cms/plugin-forms@2313

@emdash-cms/plugin-webhook-notifier

npm i https://pkg.pr.new/@emdash-cms/plugin-webhook-notifier@2313

commit: cacf517

[github-actions]

Overlapping PRs

This PR modifies files that are also changed by other open PRs:

• fix: guard FTS triggers so only real content changes re-tokenize #2314 (8 shared files)
• feat(core): configurable titleField and dateField per collection #1973 (3 shared files)

This may cause merge conflicts or duplicated work. A maintainer will coordinate.

[ascorbic]

Reviewed all three of these (#2313, #2314, #2391) – great work. Since they're now coupled by migration order, here's the plan to land them all:

Merge 060 into 059. as separate migrations, every existing site needs to rebuild the full corpus twice back to back, inside runtime init on D1. Add the WHEN clause to 059's trigger SQL here and drop migration 060 from #2314 entirely. That also removes 060's failure mode where the repopulate hard-fails init if the FTS table is missing, and the lost-update heal (no drop/create window of its own any more). #2314 keeps the fts-manager change and the write-amplification tests.

Fixes on this PR before merge:

• The json_valid guard drops legacy scalar values that happen to parse as JSON: a portableText column holding 2024 or "Some title" is searchable today and becomes silently unsearchable after this. Filter on shape instead: json_type(x) IN ('array','object') — in both the manager and the migration, plus a test row for a scalar value.
• The comment saying the SQL extraction mirrors extractPlainText isn't quite right — the SQL is a superset (any-depth text/alt/caption/code regardless of _type). This is fine, but extractPlainText is what vectorize/ai-search index with, so the two now see different text for the same document. Fix the comment to say superset; actually reconciling them can be a follow-up.
• Inline the tokenizer allowlist in the migration rather than importing SEARCH_TOKENIZERS. a replay of this migration after that constant changes would silently reset a site's tokenizer, which is the same class of bug this PR is fixing for 039. You already hardcode the fallback in three places, so this just makes it consistent.

On #2314: with 060 gone it's the manager change + tests. In the tests, the publish-path test writes identical bytes back, so it can't fail by construction — have it call repo.publish() for real, so "publish doesn't re-tokenize" properly tested

On #2391: renumber 061 to 060 once you've merged 060 into 059, and base that branch on #2314's the way #2314 sits on #2313. Optional but free: widen the index to (created_at, id) so the residual sort disappears entirely

I'll merge the three bottom-up with a rebase between each.

[edrpls]

All five points from the landing plan are in — thanks for the thorough pass, @ascorbic.

On this PR (four new commits on top of the rebase):

• WHEN guard folded into 059 (fix: fold the update-trigger WHEN guard into the rebuild migration): the rebuild now emits the guarded update trigger directly, so an upgrade re-tokenizes the corpus once. Added a migration-level test asserting a metadata-only UPDATE leaves the FTS segment bytes untouched after the rebuild (fails without the guard).
• Scalar fix: extraction is now gated on json_valid(x) AND json_type(x) IN ('array','object') in both the manager and the migration. New tests cover a 2024 integer scalar and a quoted-string scalar surviving as searchable, through both the trigger path and the migration rebuild — both fail on the old json_valid-only guard.
• Tokenizer allowlist frozen: the migration now carries its own KNOWN_TOKENIZERS copy instead of importing SEARCH_TOKENIZERS.
• Superset comment: the searchValueExpr docblock now states the SQL extraction is a superset of extractPlainText and that search and the extractPlainText consumers (vectorize/ai-search) can see different text; reconciling them stays a follow-up.

On #2314: migration 060 is gone (file, registration, its test suite); the branch is the manager change + write-amplification tests. The publish test now calls repo.publish() for real — verified it fails when the guard is stripped.

On #2391: renumbered to 060_media_usage_created_at_index, restacked on #2314's branch the way #2314 sits on this one, and the index widened to (created_at, id) to match the GC batches' ordering.

All three branches: lint clean, full search/migration suites green, typecheck green across packages. Ready for the bottom-up merge.

[edrpls]

Heads-up before the bottom-up merge: #2407 took migration slot 059 on main, so the chain renumbered once more — this PR's migration is now 060_fts_plain_text and #2391's is 061_media_usage_created_at_index (#2314 still ships no migration). All three branches re-stacked on latest main, suites green.

[edrpls]

Renumbered again: #2194 took slot 060 on main, so this PR's migration is now 061_fts_plain_text and #2391's is 062_media_usage_created_at_index (#2314 unchanged, no migration). Chain re-stacked on latest main, suites green. That's the sixth renumber — happy to keep absorbing them, but merging bottom-up soon would end the treadmill 🙂

[edrpls]

Renumbered after #2324/#2394 took slots 061–063: this PR's migration is now 064_fts_plain_text (#2314 unchanged, no migration). #2391 is closed — #2324's idx__emdash_media_usage_cleanup_scan supersedes it — so the landing chain is down to #2313 → #2314. Both re-stacked on latest main, suites green.

[edrpls]

Status note on the red Smoke Tests check: it's Google Fonts, not this PR. Three smoke/E2E-adjacent build failures across this chain in the past day are all Astro CannotFetchFontFile with a 404 from fonts.gstatic.com, each killing a different template build:

• run 31515086406 — JetBrains Mono, templates/blog
• run 31522133287 — JetBrains Mono, templates/blog-cloudflare
• run 31523985173 — Playfair Display, templates/portfolio

Identical trees pass between failures (#2314 is fully green right now on the same base), so it's transient CDN 404s with no retry in the font fetch path — any PR's smoke run can lose this roulette. I've stopped re-triggering to avoid burning CI; happy to re-run once more on request, or this may be worth a CI-side mitigation (retry/backoff or vendored fonts for template builds). Everything code-related on this PR is green.

[ascorbic]

Thanks