Integrate Relay for website email and retire Datamailer

[alexeygrigorev]

Parent epic: #6

Related implementation issues: #48, #49, #50
Recorded owner decision: #21

Outcome

Use DataTalksClub/relay as the sole template and transactional-email delivery service for the unified website. The website records the business intent and a redacted delivery projection, manages Relay-owned templates through Studio/admin API, and never implements a second SES sender. Datamailer is migration-only: it may be read, frozen, drained, and reconciled, but it receives no new website sends and is retired after the cutover gates pass.

This owner decision supersedes the direct-SES/long-term-Datamailer recommendations currently present in _docs/specs/open-decisions.md and _docs/specs/05-events-registration-email.md. Updating those normative documents is part of this issue; new implementation must not follow the superseded text while that documentation change is pending.

Verified Relay baseline and gaps

The audited reference is Relay commit c0fdfac632f057f1e6a467111613928c4899b13b, deployed only to the development sandbox at https://relay.dtcdev.click. There is no Relay production deployment or production action in scope.

That reference already provides tenant Bearer authentication and these useful starting points:

• GET/PUT /api/transactional/templates/{template_key};
• POST /api/transactional/send, including dry_run;
• GET /api/transactional/messages/{message_id};
• a client/idempotency uniqueness constraint, rendered message snapshots, SES event ingestion, suppression data, and a CMP-specific callback outbox.

It does not yet satisfy the website contract. The implementation must treat these as blocking Relay hardening, not undocumented assumptions:

• templates are one mutable row with no immutable published version;
• rendering uses autoescape=False and does not provide the required safe authoring boundary;
• the single-send idempotency key is optional, equal-key/different-request replays return the old message instead of 409, and lookup-then-insert races are not recovered deterministically;
• delivery states have no expiring lease, explicit provider-accepted/delivered distinction, retryable/dead split, or ambiguous provider acknowledgement;
• suppression is checked when the message is created but not atomically rechecked when a worker claims it;
• API keys are tenant-scoped but have no endpoint scopes or expiry;
• callbacks are CMP-specific bearer-token posts, not a generic signed tenant callback contract;
• valid unmatched provider events and missed callbacks do not yet have the required reconciliation lifecycle;
• the documented dry run must be proven zero-write, including no contact creation.

Ownership boundary

Website owns

• the business action and its transaction;
• one durable logical EmailDelivery intent with purpose, business-object references, recipient reference/snapshot according to retention policy, Relay template key and version, minimal context snapshot, and one stable idempotency key;
• after-commit submission, a redacted local projection of Relay status, callback ingestion, reconciliation scheduling, and audited operator commands;
• Studio/admin API permissions, confirmation, revision checks, and audit for template-management and delivery-management actions.

Relay owns

• the canonical template key, drafts, immutable published versions, safe validation/rendering, bootstrap template content, sender policy, and the rendered subject/plain/HTML snapshot used for a message;
• provider submission, credentials/identities, queues/workers, leases/attempts/backoff, SES event ingestion, suppression, provider diagnostics, callbacks, reconciliation, and authoritative transport status;
• the versioned client API/OpenAPI contract and tenant/scoped credential enforcement.

The website's Studio and admin API manage/proxy Relay templates. They do not create a website EmailTemplate source of truth or copy Relay workers into Django. A local cache may hold redacted display metadata for availability but can never be edited or used to render/send when Relay is unavailable.

New website code must not invoke Datamailer or SES. SES region, identities, configuration sets, and provider event plumbing are Relay/infrastructure implementation details. Course/CMP messages use Relay sender ID courses, resolving in development to DataTalks.Club Courses <courses@dtcdev.click> and reusing the existing dtcdev.click identity. Other real sender/reply-to mappings fail closed until their purpose is approved in #22 and configured in Relay; there is no arbitrary fallback sender.

Required Relay contract milestone

These Relay capabilities must be deployed and contract-tested in the sandbox before the website real-send path is enabled.

Credentials

• Provision separate tenant keys for website-runtime (transactional:send, transactional:read, templates:read) and website-template-admin (templates:read, templates:write, transactional:dry_run).
• Keys store only hashes, carry an explicit expiry, can overlap during rotation, are immediately revocable, and deny every undeclared endpoint. The callback signing secret is separate from API keys.
• Secrets live in the approved secrets store and never appear in source, screenshots, API responses, ordinary logs, or audit metadata.

Relay-owned versioned templates

• A template key has at most one editable draft and monotonically numbered immutable published versions.
• GET /api/transactional/templates/{key} returns redacted catalog metadata, draft revision, and current published-version reference.
• PUT /api/transactional/templates/{key}/draft creates/updates only the draft and requires expected_revision; stale revision returns 409 without mutation.
• POST /api/transactional/templates/{key}/publish requires draft revision plus idempotency key and creates a new immutable published version. Publishing an older version for rollback creates a new version copied from it; no historical version is mutated.
• GET /api/transactional/templates/{key}/versions/{version} returns that immutable version.
• POST /api/transactional/templates/{key}/render renders a named draft revision or published version without send or persistence side effects. Real/test sends use the same validation and render implementation.
• Existing mutable PUT /api/transactional/templates/{key} must not remain a path that can mutate published content for the website credential; retain only a documented migration behavior or reject it.
• HTML variables escape by default; authored rich content is sanitized to an allowlist; arbitrary template tags/filters and header injection are rejected; plain/HTML meaning and links are validated. Context schema validates missing, extra, and wrong-type values.

Every send names both template_key and immutable template_version. Relay snapshots the version, resolved sender, recipient, subject, plain body, HTML body, and delivery-affecting options before queueing, so later draft/publish/rollback actions cannot change queued output.

Idempotent submission and state

• POST /api/transactional/send requires a caller idempotency key. Relay stores a SHA-256 canonical request hash over every delivery-affecting field, including tenant, normalized recipient, purpose, template key/version, context, sender/reply-to, CC/BCC, headers, and message parts.
• Same tenant/key plus the same hash returns the original Relay message and marks the response as an idempotent replay. Same tenant/key with a different hash returns 409 idempotency_conflict and creates/enqueues nothing.
• Concurrent first submissions converge through the database uniqueness constraint: the losing insert reloads and compares the winner rather than leaking an integrity error or creating a second queue item.
• Relay's authoritative state set is queued, leased, provider_accepted, delivered, retryable, ambiguous, suppressed, dead, hard_bounced, and complained. Each state has guarded transitions and timestamps; accepted is never displayed as delivered.
• Claims use an owner/token plus expiry. Worker death makes an unsubmitted expired lease safely claimable. An uncertain loss after the provider request enters ambiguous, is not automatically resent, and requires provider-event/status reconciliation or explicit audited resolution.
• Suppression and sender eligibility are rechecked under the claim transaction immediately before provider submission. A newly bounced/complained recipient is suppressed even if the message was queued earlier.
• Attempts record claim, provider correlation, outcome, redacted error, and timing. Backoff is bounded; permanent failures become dead.

Callbacks and reconciliation

• Relay publishes a generic tenant callback for every transport transition needed by the website projection. Its redacted payload includes schema version, stable event ID, event type/status, occurred time, Relay message ID, website idempotency/correlation key, template key/version, and safe reason code—never body, API key, or full provider payload.
• Callbacks use the Relay HMAC convention over <timestamp>.<raw-body> with X-Relay-Timestamp and X-Relay-Signature, have a five-minute replay window, use a per-client secret, and are stored/retried from a Relay outbox. The website deduplicates event IDs and applies reordered events through the guarded state machine.
• GET /api/transactional/messages/{message_id} returns authoritative current state and a redacted ordered event timeline. Replaying the exact send request is the lookup/recovery path when an HTTP timeout occurred before the website stored the message ID.
• Valid unmatched provider events are retained redacted for seven days and retried for correlation; they become operator-visible if still unmatched. Provider event ID deduplication is mandatory.
• The website reconciles due nonterminal projections at least every five minutes with bounded batches/backoff, and rechecks recently terminal deliveries daily for seven days so a missed late bounce/complaint converges. Manual reconciliation of one delivery uses the same service.

Website integration

1. Business state and one unique local EmailDelivery intent commit in the same database transaction. Network activity is forbidden inside that transaction.
2. After commit, a durable website job submits the pinned Relay request. Retry uses the same key and byte-equivalent semantic request; it never creates a new logical intent.
3. The website stores Relay message ID, authoritative-status projection, last reconciled time, safe reason, and redacted attempt/callback audit. Relay remains the transport source of truth.
4. The local projection represents pending submission, submitted Relay states, callback/reconciliation freshness, and an explicit integration error; it must not invent provider delivery.
5. Studio/admin API provide capability parity for redacted inspect, refresh/reconcile, retry of a safe pre-provider/retryable failure, ambiguous resolution, and manual resend. Manual resend creates a new local intent/key linked to the original and requires permission, fresh revision, explicit confirmation, reason, and audit.
6. Template operations proxy the Relay contract through shared website application services. Relay unavailability produces a safe, retryable UI/API error and never causes local template divergence.

Datamailer migration and one-sender cutover

#50 owns the migration mechanics:

1. inventory Datamailer history, pending work, workers, endpoints, and every caller;
2. import history and ID mappings with all outbound paths disabled, then compare counts/checksums and exceptions;
3. deploy/prove Relay sandbox contract, allowlist/simulation, callback, reconciliation, and a controlled courses canary;
4. freeze new Datamailer intake and drain or explicitly classify every old pending item;
5. prove one active sender per message purpose, then enable new website Relay submissions;
6. observe and reconcile before disabling/removing compatibility access.

The Datamailer adapter is read-only/import/reconciliation code. It cannot submit new website messages. After the one-way cutover gate, rollback holds new website intents and reconciles Relay; it does not silently re-enable Datamailer or run two senders. Do not delete Datamailer data/infrastructure until retention, evidence, rollback-window, and owner gates pass.

Scope by issue

• Decision: Use Relay for transactional email and retire Datamailer #21 records the Relay/provider, sender, production, and ambiguity decisions; owner input is no longer pending.
• Manage Relay-owned versioned transactional templates in Studio #48 implements the Relay-owned versioned-template contract and the website Studio/admin proxy, safe preview/test/publish/rollback UX, permissions, revision, and audit.
• Build the durable Relay delivery intent and status lifecycle #49 implements Relay hardening for idempotency/state/leases/suppression/callback/reconciliation plus the website local intent, adapter, projection, and delivery operations.
• Wire approved messages through Relay and retire Datamailer #50 registers approved purposes and performs Datamailer inventory/import/freeze/drain/cutover/retirement.
• Decision: Confirm the MVP transactional email purpose catalog #22 supplies the complete purpose/sender/reply-to/retention catalog. Purposes not yet approved cannot real-send.
• AWS and secret changes use the infrastructure repository and Epic: Provision portable AWS infrastructure with Terraform #9 boundaries, development first. This issue grants no production authority.

Acceptance criteria

• Decision: Use Relay for transactional email and retire Datamailer #21 and the normative email/open-decision specifications record Relay as the template/delivery owner, prohibit website direct SES and new Datamailer sends, and mark production sender/deployment as deferred.
• Relay sandbox identifies the exact deployed commit/OpenAPI version and passes a website contract suite before real send is enabled.
• Scoped, expiring runtime/template-admin credentials and a separate callback secret are provisioned, rotation/revocation is tested, and undeclared scopes are denied.
• Relay templates have draft revision plus immutable published versions; publish/rollback never mutates a version and queued output is pinned byte-for-byte.
• Relay render/preview/dry-run is safe and zero-write, escapes variables by default, sanitizes rich content, rejects unsupported template code/header injection, and matches real-send validation/rendering.
• Mandatory request-hash idempotency returns the original for an exact replay, returns 409 for changed work, and converges concurrent first submissions to one Relay message/queue item.
• Business state and one local logical delivery intent commit atomically; Relay submission occurs after commit and uses the same stable key through timeout/retry recovery.
• Lease expiry, before/during/after-provider crash cases, explicit ambiguity, guarded transitions, backoff/dead state, and accepted-versus-delivered semantics pass failure injection without blind duplicate sends.
• Suppression is rechecked at claim; bounce/complaint and valid unmatched events converge through signed callbacks or reconciliation and prevent later sends where policy requires.
• Generic callbacks are tenant-scoped, HMAC/timestamp verified, deduplicated, reorder-tolerant, redacted, durably retried, and recoverable by the scheduled/manual reconciliation paths.
• Studio/admin API template and delivery capabilities have parity, least-privilege roles, revision/idempotency/confirmation/audit controls, masked PII, and safe Relay-unavailable states.
• courses resolves only to the approved development sender; unknown purpose/sender mapping and non-allowlisted development recipients fail closed.
• Datamailer import sends nothing, count/checksum exceptions are explicit, freeze/drain inventory reaches zero/classified, and evidence proves only one active sender per purpose.
• Contract, migration, unit, concurrency, failure-injection, security, and Playwright suites pass; metrics/alerts/runbooks cover queue age, expired leases, ambiguity, callback lag/failure, reconciliation lag, suppression, terminal failure, and cutover rollback.

Mandatory validation scenarios

1. Event/course business transaction rollback, commit, trigger replay, concurrent submit, Relay timeout before response, and response loss after Relay commit all converge to one website intent and one Relay message.
2. Same idempotency key with changed recipient/template version/context/sender/options returns 409; exact replay returns the original message, including under an insert race.
3. Crash before claim, after claim, before provider call, during uncertain provider acknowledgement, after provider acceptance, and after callback enqueue exercises lease recovery and ambiguity without blind resend.
4. A complaint arrives after queue but before claim; claim suppresses. Duplicate/out-of-order/missing/invalid-signature callback and late bounce all converge through reconciliation without status regression.
5. Edit/publish/rollback while a message is queued; its template version, sender, subject, plain body, and HTML body remain unchanged. Malicious context/source and unsupported template constructs are safely rejected or escaped.
6. Expired/revoked/wrong-scope/cross-tenant credentials and callback signatures fail without leaking whether a message/template exists.
7. Dry-run/preview creates no contact, message, event, queue item, or provider call; a permitted sandbox canary reaches accepted then delivered with redacted evidence.
8. Import duplicate/missing/unknown Datamailer IDs while sending is disabled; freeze/drain/cutover rehearsal proves no new Datamailer intake and no dual sender.

Playwright acceptance

• Manage a Relay-backed template draft, stale edit, preview, publish, republish-for-rollback, and test/dry-run through Studio at desktop and mobile widths; inspect safe rendering, plain/HTML parity, focus/errors, and confirmation screenshots.
• Inspect queued, leased, provider-accepted, delivered, retryable, ambiguous, suppressed, dead, bounced, and complained delivery projections with masked recipient/provider details.
• Exercise permitted reconcile/retry/ambiguous-resolution/manual-resend and negative role/stale-revision/double-submit paths in both Studio and admin API parity tests.
• Stop or fault Relay and verify safe unavailable/lagging states rather than local fallback rendering or direct sending.

Non-goals

• Marketing/newsletter campaigns or tracking pixels.
• A website-owned template content store, renderer, SES adapter, queue worker, or provider event stack.
• New website sends through Datamailer, dual-send fallback, or deleting legacy records before the migration gates.
• Relay production deployment, production DNS/sender approval, or broad infrastructure mutation.
• Exposing secrets, full recipient addresses, message bodies, or raw provider payloads in normal logs/UI/evidence.
• Treating provider acceptance as delivery or automatically retrying an ambiguous provider call.