fix(core): reconcile media usage indexes automatically by khoinguyenpham04 · Pull Request #2443 · emdash-cms/emdash

khoinguyenpham04 · 2026-08-12T17:25:26Z

What does this PR do?

Makes Media Usage coverage recover automatically and resumably after activation.

This PR:

adds durable per-collection reconciliation state with leases, retries, epochs, and crash-safe checkpoints;
scans historical content and canonical projection sources in bounded 50-row pages;
prevents stale workers and changed content from publishing obsolete projections;
completes coverage only after guarded work and source barriers are empty;
schedules fair, bounded entry work, collection cleanup, and reconciliation lanes on Cloudflare and Node;
preserves V1 behavior, merged collection-deletion safety from fix(core): make media usage collection deletion crash-safe #2433, and logged-out query counts;
keeps incremental capture fail-closed and does not activate it automatically.

Builds on merged #2433.

Type of change

Checklist

I have read CONTRIBUTING.md
pnpm typecheck passes
pnpm lint passes
pnpm test passes (targeted tests for this change)
pnpm format has been run
I have added/updated tests for my changes (if applicable)
User-visible strings in the admin UI are wrapped for translation (not applicable; no admin UI or user-facing UI strings)
I have added a changeset (if this PR changes a published package)
New features link to an approved Discussion: not applicable; this repairs and completes the existing durable Media Usage indexing system

AI-generated code disclosure

This PR includes AI-generated code — model/tool: OpenAI Codex (GPT-5) with GPT-5.6 Terra adversarial review

Screenshots / test output

No visual changes.

Verified after integrating merged #2433:

core and Cloudflare package typechecks;
quick lint and targeted formatting checks;
focused Media Usage, migration, API, authorization, and scheduler suites: 490 passed, 2 skipped;
real D1/workerd deletion and projection suites: 7 passed;
Cloudflare configuration and scheduled-handler suites: 7 passed;
final diff against main: 40 files, 3,445 additions, 74 deletions.

PostgreSQL runtime tests were not run because EMDASH_TEST_PG is not configured locally.

Try this PR

Open a fresh playground →

A full working EmDash site, deployed from this branch. Each visit gets its own session-scoped sandbox: no login needed and no shared state. Try the admin, edit content, hit the public site.

_{Tracks feature/media-usage-automatic-reconciliation. Updated automatically when the playground redeploys.}

…utomatic-reconciliation # Conflicts: # .changeset/safe-collection-deletion.md # packages/core/src/database/migrations/runner.ts # packages/core/src/database/repositories/media-usage.ts # packages/core/src/database/types.ts # packages/core/src/emdash-runtime.ts # packages/core/src/media/usage/collection-deletion-processor.ts # packages/core/src/media/usage/collection-deletion.ts # packages/core/tests/integration/database/media-usage-incremental-work-migration.test.ts # packages/core/tests/integration/database/migrations.test.ts # packages/core/tests/integration/runtime/media-usage-scheduled-driver.test.ts # packages/core/tests/workerd/media-usage-collection-deletion-d1.test.ts

changeset-bot · 2026-08-12T17:25:31Z

🦋 Changeset detected

Latest commit: 29ae696

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 17 packages

Name	Type
emdash	Patch
@emdash-cms/cloudflare	Patch
@emdash-cms/sandbox-workerd	Patch
@emdash-cms/plugin-mcp-smoke	Patch
@emdash-cms/fixture-perf-site	Patch
@emdash-cms/perf-demo-site	Patch
@emdash-cms/cache-demo-site	Patch
@emdash-cms/do-demo-site	Patch
@emdash-cms/do-solo-demo-site	Patch
@emdash-cms/admin	Patch
@emdash-cms/auth	Patch
@emdash-cms/blocks	Patch
@emdash-cms/gutenberg-to-portable-text	Patch
@emdash-cms/x402	Patch
create-emdash	Patch
@emdash-cms/auth-atproto	Patch
@emdash-cms/plugin-embeds	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

github-actions · 2026-08-12T17:25:40Z

Scope check

This PR changes 3,519 lines across 40 files. Large PRs are harder to review and more likely to be closed without review.
This PR spans 4 different areas (area/core, area/docs, area/templates, area/cloudflare). Consider breaking it into smaller, focused PRs.

If this scope is intentional, no action needed. A maintainer will review it. If not, please consider splitting this into smaller PRs.

See CONTRIBUTING.md for contribution guidelines.

cloudflare-workers-and-pages · 2026-08-12T17:25:47Z

Deploying with Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Updated (UTC)
✅ Deployment successful! View logs	docs	`7a3ff56`	Aug 13 2026, 10:50 AM

cloudflare-workers-and-pages · 2026-08-12T17:26:37Z

Deploying with Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Updated (UTC)
✅ Deployment successful! View logs	emdash-demo-do	`29ae696`	Aug 13 2026, 11:18 AM

cloudflare-workers-and-pages · 2026-08-12T17:27:21Z

Deploying with Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Updated (UTC)
✅ Deployment successful! View logs	emdash-playground	`29ae696`	Aug 13 2026, 11:19 AM

pkg-pr-new · 2026-08-12T17:28:21Z

Open in StackBlitz

@emdash-cms/admin

npm i https://pkg.pr.new/@emdash-cms/admin@2443

@emdash-cms/auth

npm i https://pkg.pr.new/@emdash-cms/auth@2443

@emdash-cms/auth-atproto

npm i https://pkg.pr.new/@emdash-cms/auth-atproto@2443

@emdash-cms/blocks

npm i https://pkg.pr.new/@emdash-cms/blocks@2443

@emdash-cms/cloudflare

npm i https://pkg.pr.new/@emdash-cms/cloudflare@2443

@emdash-cms/contentful-to-portable-text

npm i https://pkg.pr.new/@emdash-cms/contentful-to-portable-text@2443

emdash

npm i https://pkg.pr.new/emdash@2443

create-emdash

npm i https://pkg.pr.new/create-emdash@2443

@emdash-cms/gutenberg-to-portable-text

npm i https://pkg.pr.new/@emdash-cms/gutenberg-to-portable-text@2443

@emdash-cms/plugin-cli

npm i https://pkg.pr.new/@emdash-cms/plugin-cli@2443

@emdash-cms/plugin-types

npm i https://pkg.pr.new/@emdash-cms/plugin-types@2443

@emdash-cms/registry-client

npm i https://pkg.pr.new/@emdash-cms/registry-client@2443

@emdash-cms/registry-lexicons

npm i https://pkg.pr.new/@emdash-cms/registry-lexicons@2443

@emdash-cms/registry-verification

npm i https://pkg.pr.new/@emdash-cms/registry-verification@2443

@emdash-cms/sandbox-workerd

npm i https://pkg.pr.new/@emdash-cms/sandbox-workerd@2443

@emdash-cms/x402

npm i https://pkg.pr.new/@emdash-cms/x402@2443

@emdash-cms/plugin-ai-moderation

npm i https://pkg.pr.new/@emdash-cms/plugin-ai-moderation@2443

@emdash-cms/plugin-atproto

npm i https://pkg.pr.new/@emdash-cms/plugin-atproto@2443

@emdash-cms/plugin-audit-log

npm i https://pkg.pr.new/@emdash-cms/plugin-audit-log@2443

@emdash-cms/plugin-color

npm i https://pkg.pr.new/@emdash-cms/plugin-color@2443

@emdash-cms/plugin-embeds

npm i https://pkg.pr.new/@emdash-cms/plugin-embeds@2443

@emdash-cms/plugin-field-kit

npm i https://pkg.pr.new/@emdash-cms/plugin-field-kit@2443

@emdash-cms/plugin-forms

npm i https://pkg.pr.new/@emdash-cms/plugin-forms@2443

@emdash-cms/plugin-webhook-notifier

npm i https://pkg.pr.new/@emdash-cms/plugin-webhook-notifier@2443

commit: 29ae696

emdashbot

This PR completes the durable Media Usage indexing system by adding automatic, resumable, bounded background reconciliation. After reading the diff, the key implementation files, and tracing the scheduling/runtime wiring, the approach looks sound and well-fitted to EmDash: it reuses the existing lease/epoch/work machinery, keeps scans bounded to 50-row pages, fences stale claims, does not activate incremental capture automatically, and routes the new lane through the same Cloudflare Cron Trigger / Node scheduler paths already used for maintenance.

I checked:

SQL safety: all dynamic table/identifier references go through validateIdentifier() and Kysely sql.ref(); embedded fragments use Kysely template composition; no sql.raw() string interpolation.
Migration discipline: 066_media_usage_reconciliation.ts is sequential, registered in runner.ts, creates indexes on the new coordinator table and on _emdash_media_usage_index_status for the reconciliation gate, and its down() guards against rollback after activation.
Locale correctness: the new content-table scans paginate by content.id and only capture ID boundaries/source identity; locale is stored per-source via the existing _emdash_media_usage_sources row, so no content query forgets the per-locale model.
Logged-out hot path: reconciliation/work is only driven by the Cloudflare scheduled() handler or the Node timer scheduler, not by request handlers or middleware; no new queries are added to public page renders.
Cloudflare scheduling: createScheduledHandler() remains backwards-compatible (no options = old behavior); the templated configuration correctly splits general and Media Usage cron expressions and validates they differ.
Comments / AGENTS conventions: no PR/issue references, narrative, or justification comments in the new code.
Tests: new integration and Cloudflare worker scheduled-handler tests exercise observable behavior (lane dispatch, claim fencing, scan/source finalization) rather than asserting implementation details back at themselves.

No blocking issues. Good to merge.

github-actions · 2026-08-12T18:25:34Z

Overlapping PRs

This PR modifies files that are also changed by other open PRs:

Add deployment-managed core migration primitives #2437 (4 shared files)
feat(core): add getBylines() for listing a site's bylines #2352 (3 shared files)
feat(content): filter indexed custom fields #2213 (4 shared files)
feat(content): support indexed custom field sorting #2212 (3 shared files)
feat(core): configurable titleField and dateField per collection #1973 (4 shared files)
feat(media): show where media is used #2225 (4 shared files)
feat: gate sandboxed plugins behind Worker Loader so free-tier Cloudflare deploys work #2351 (4 shared files)

This may cause merge conflicts or duplicated work. A maintainer will coordinate.

emdashbot

This PR completes the durable Media Usage indexing system by adding automatic, resumable, bounded background reconciliation. Re-reading the diff after the prior review, the approach still looks sound and well-fitted to EmDash: it reuses the existing lease/epoch/work machinery, keeps scans bounded to 50-row pages, fences stale claims, does not activate incremental capture automatically, and correctly routes the new lane through both Cloudflare Cron Trigger and Node timer scheduler paths.

I re-checked:

SQL safety: all dynamic table/identifier references go through validateIdentifier() and Kysely sql.ref(); embedded fragments use Kysely template composition.
Migration discipline: 066_media_usage_reconciliation.ts is sequential, registered in runner.ts, adds the required indexes, and its down() guards against rollback after activation.
Locale correctness: the new content-table scans paginate by content.id; each source row stores its own locale, so no content query forgets the per-locale model.
Logged-out hot path: reconciliation/work is only driven by scheduled handlers, not by request handlers or middleware.
Cloudflare scheduling: createScheduledHandler() remains backwards-compatible, validates that general and Media Usage cron expressions differ, and dispatches exactly one lane per trigger.
AGENTS conventions: no narrative/issue-reference/justification comments; no untranslated UI strings; changeset is user-facing and correct.
Tests: the new integration and scheduled-driver tests exercise observable behavior (lane dispatch, claim fencing, scan/source finalization, epoch restart) rather than asserting implementation details back at themselves.

No blockers, regressions, or convention violations were found. This re-review confirms the prior approval; the branch is ready to merge.

ascorbic

Great, thanks

khoinguyenpham04 added 17 commits August 12, 2026 11:45

feat(core): add guarded collection deletion foundation

ba4646a

feat(core): safely detach activated collections

302ac9e

feat(core): process bounded collection deletion cleanup

a576d10

feat(core): expose collection deletion recovery controls

83dbf0c

fix(cloudflare): harden collection deletion guards

2d36e13

fix(core): bound collection deletion completion checks

725ef86

fix(core): use database time for deletion progress

110f945

chore(core): register collection deletion schemas

1d81a96

chore(core): keep deletion lease guards transaction-scoped

808d03e

fix(core): preserve collection deletion compatibility

26e4b26

fix(cloudflare): pin mutation reads to DO primary

90f01f8

feat(media): add reconciliation coordinator state

e2a18b6

feat(media): add bounded reconciliation scan

136ece4

feat(media): finalize automatic reconciliation

e3d35a2

feat(media): schedule automatic reconciliation

da3101d

fix(media): preserve scheduled maintenance compatibility

5ce7f25

github-actions Bot added area/cloudflare area/core area/docs area/templates size/XL labels Aug 12, 2026

khoinguyenpham04 marked this pull request as ready for review August 12, 2026 17:26

khoinguyenpham04 requested a review from ascorbic August 12, 2026 17:27

github-actions Bot added the review/needs-review No maintainer or bot review yet label Aug 12, 2026

emdashbot Bot removed the bot:review Trigger an emdashbot code review on this PR label Aug 12, 2026

khoinguyenpham04 added the bot:review Trigger an emdashbot code review on this PR label Aug 12, 2026

emdashbot Bot approved these changes Aug 12, 2026

View reviewed changes

emdashbot Bot removed the bot:review Trigger an emdashbot code review on this PR label Aug 12, 2026

github-actions Bot added review/approved Approved; no new commits since overlap and removed review/needs-review No maintainer or bot review yet labels Aug 12, 2026

github-actions Bot mentioned this pull request Aug 12, 2026

Add deployment-managed core migration primitives #2437

Open

18 tasks

khoinguyenpham04 mentioned this pull request Aug 12, 2026

fix(core): add controlled media usage activation #2445

Open

18 tasks

fix(core): guard null media usage revisions on postgres

1a96dda

github-actions Bot added review/needs-rereview Author pushed changes since the last review and removed review/approved Approved; no new commits since labels Aug 12, 2026

ascorbic reviewed Aug 13, 2026

View reviewed changes

Comment thread docs/src/content/docs/deployment/cloudflare.mdx Outdated

Comment thread packages/cloudflare/src/worker.ts Outdated

Comment thread packages/cloudflare/src/worker.ts Outdated

github-actions Bot added review/approved Approved; no new commits since and removed review/needs-rereview Author pushed changes since the last review labels Aug 13, 2026

fix(cloudflare): default scheduled cron routing

7a3ff56

github-actions Bot added review/needs-rereview Author pushed changes since the last review and removed review/approved Approved; no new commits since labels Aug 13, 2026

docs(cloudflare): keep worker comments current

29ae696

khoinguyenpham04 added the bot:review Trigger an emdashbot code review on this PR label Aug 13, 2026

emdashbot Bot approved these changes Aug 13, 2026

View reviewed changes

emdashbot Bot removed the bot:review Trigger an emdashbot code review on this PR label Aug 13, 2026

ascorbic approved these changes Aug 13, 2026

View reviewed changes

github-actions Bot added review/approved Approved; no new commits since and removed review/needs-rereview Author pushed changes since the last review labels Aug 13, 2026

khoinguyenpham04 merged commit 3ceabc4 into main Aug 13, 2026
52 checks passed

khoinguyenpham04 deleted the feature/media-usage-automatic-reconciliation branch August 13, 2026 12:29

emdashbot Bot mentioned this pull request Aug 13, 2026

ci: release #2419

Open

Navigation Menu

Uh oh!

fix(core): reconcile media usage indexes automatically - #2443

fix(core): reconcile media usage indexes automatically#2443
khoinguyenpham04 merged 21 commits into
mainfrom
feature/media-usage-automatic-reconciliation

khoinguyenpham04 commented Aug 12, 2026 •

edited by github-actions Bot

Loading

Uh oh!

changeset-bot Bot commented Aug 12, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 12, 2026

Uh oh!

cloudflare-workers-and-pages Bot commented Aug 12, 2026 •

edited

Loading

Uh oh!

cloudflare-workers-and-pages Bot commented Aug 12, 2026 •

edited

Loading

Uh oh!

cloudflare-workers-and-pages Bot commented Aug 12, 2026 •

edited

Loading

Uh oh!

pkg-pr-new Bot commented Aug 12, 2026 •

edited

Loading

Uh oh!

emdashbot Bot left a comment

Uh oh!

github-actions Bot commented Aug 12, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

emdashbot Bot left a comment

Uh oh!

ascorbic left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

khoinguyenpham04 commented Aug 12, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Type of change

Checklist

AI-generated code disclosure

Screenshots / test output

Try this PR

Uh oh!

changeset-bot Bot commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

github-actions Bot commented Aug 12, 2026

Scope check

Uh oh!

cloudflare-workers-and-pages Bot commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying with Cloudflare Workers

Uh oh!

cloudflare-workers-and-pages Bot commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying with Cloudflare Workers

Uh oh!

cloudflare-workers-and-pages Bot commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying with Cloudflare Workers

Uh oh!

pkg-pr-new Bot commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

emdashbot Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Aug 12, 2026

Overlapping PRs

Uh oh!

Uh oh!

Uh oh!

Uh oh!

emdashbot Bot left a comment

Choose a reason for hiding this comment

Uh oh!

ascorbic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

khoinguyenpham04 commented Aug 12, 2026 •

edited by github-actions Bot

Loading

changeset-bot Bot commented Aug 12, 2026 •

edited

Loading

cloudflare-workers-and-pages Bot commented Aug 12, 2026 •

edited

Loading

cloudflare-workers-and-pages Bot commented Aug 12, 2026 •

edited

Loading

cloudflare-workers-and-pages Bot commented Aug 12, 2026 •

edited

Loading

pkg-pr-new Bot commented Aug 12, 2026 •

edited

Loading