From 3de628c6718ac641956da5599721bd03b3eded6d Mon Sep 17 00:00:00 2001 From: Mrugesh Mohapatra Date: Mon, 17 Aug 2026 09:29:03 +0530 Subject: [PATCH 01/41] docs: design the fix for drift at its source --- docs/ONBOARDING.md | 221 ++++++++++++++++++++++++++++ docs/design/0005-drift-at-source.md | 187 +++++++++++++++++++++++ 2 files changed, 408 insertions(+) create mode 100644 docs/ONBOARDING.md create mode 100644 docs/design/0005-drift-at-source.md diff --git a/docs/ONBOARDING.md b/docs/ONBOARDING.md new file mode 100644 index 0000000..aa2f4b6 --- /dev/null +++ b/docs/ONBOARDING.md @@ -0,0 +1,221 @@ +# artemis, from scratch + +Written for someone who has never opened this repo. Read it top to bottom once; the order is deliberate, because each section depends on the one before it. + +Every claim cites `file:line`. Where a claim is not verified, it says so. + +______________________________________________________________________ + +## 1. What this service is + +artemis is a deploy proxy for static sites. Staff push a folder of files; artemis puts those files in Cloudflare R2 and points a name at them. A separate serve plane (Caddy) reads that name and serves the bytes. artemis never serves site content itself. + +It exists because R2 admin credentials must live in exactly one place. artemis holds them. Everyone else asks artemis. + +Three things it owns: + +1. **The bytes** — deploy folders in R2. +1. **The pointer** — which deploy is currently "production" or "preview" for a site. +1. **The index** — a Postgres mirror of both, so it can answer questions and clean up. + +Everything difficult in this codebase comes from the fact that those three things live in two different systems and can disagree. + +______________________________________________________________________ + +## 2. The one thing that will confuse you: two names for every site + +A site has **two different names**, in two different keyspaces, and they are not the same string in production. + +| name | example | lives in | +| --------------- | -------------------- | ---------------------------------------------------------------- | +| registry slug | `test` | the `sites` table, URLs, the CLI, JWT claims | +| storage dirname | `test.freecode.camp` | R2 keys, and `deploys.site` / `aliases.site` / `tombstones.site` | + +The conversion is one function: `handler.DeployPrefixTemplate.SiteDirname()` (`internal/handler/deploykey.go:56`). It is defined as "everything before the first `/` of the rendered deploy prefix". + +Production sets `DEPLOY_PREFIX_FORMAT=".freecode.camp/deploys/-/"`, so `SiteDirname("test")` is `test.freecode.camp`. + +**Why this is a trap.** The default format in code and in most tests is `"/deploys/-/"`. Under that format slug and dirname are *the same string*, so any code that confuses them still works — in tests, and only in tests. Every keyspace bug in this service's history has hidden exactly there, and `docs/ARCHITECTURE.md:349` says so explicitly. + +Rule of thumb when reading a function: **HTTP handlers speak slug; everything under `internal/gc` speaks dirname.** The boundary is where a handler calls `SiteDirname`. You can see both in two adjacent lines — `internal/handler/deploy.go:264` passes the *slug* to `aliasKey`, and `:266` passes the *dirname* to `withSiteLock`. + +______________________________________________________________________ + +## 3. Who is allowed to do anything + +There are exactly two credentials, and the **route decides which one applies**. There is no priority chain and no fallback, despite what ADR-016's prose suggests. + +- **GitHub bearer token** — `internal/handler/middleware.go:86`. The token is exchanged for a login via `GET /user` (`internal/auth/github.go:114`), cached ~5 minutes. Then the *handler* decides authorization per resource by intersecting the caller's GitHub teams with the teams the registry lists for that site (`internal/handler/site.go:349`). +- **Deploy-session JWT** — `internal/handler/middleware.go:130`. artemis mints it itself at `POST /api/deploy/init` (`internal/handler/deploy.go:41`), HS256, scoped to `{login, site, deployId}`, 15-minute default TTL (`internal/auth/jwt.go:34`). + +The two route groups are disjoint (`internal/server/server.go:69` vs `:102`). Uploads and finalize live on the JWT plane and make **no** GitHub call at all. + +Consequences worth internalising: + +- Team-membership changes are **not** immediate. A minted JWT keeps working until it expires. There is no re-probe on upload. +- Every team probe is made **as the caller**, using the caller's own token — that is why the raw token is kept on the request context (`middleware.go:110`). +- There are **two GitHub orgs** in play: `h.GH` for site teams, `h.RepoGH` for repo and audit teams. Confusing them is easy and silent. + +______________________________________________________________________ + +## 4. A deploy, end to end + +This is the path that produces every object you will later see in R2. + +1. `POST /api/deploy/init` — bearer auth, site-team check, server generates the `deployId` from the caller's sha, mints the JWT (`deploy.go:41-97`). +1. `PUT /api/deploy/{id}/*` — JWT auth, each file written under `DeployPrefix(site, deployId)` (`deploy.go:110-125`). Thousands of these per deploy. +1. `POST /api/deploy/{id}/finalize` — the interesting one (`deploy.go:193-300`): + 1. verify every file in the manifest actually landed in R2 (`VerifyDeployComplete`), + 1. write the marker `_artemis_meta.json` **after** that check, so the marker is never part of what was verified (`deploy.go:224` vs `:242`), + 1. take the Postgres site lock, and inside it: check the site still exists → `PUT` the alias object in R2 → write the index row in Postgres, + 1. all of step 3 runs on a **detached** context with its own 60s budget, so a client hanging up cannot abort a half-committed alias swap (`deploy.go:265`). + +**The marker is the whole ballgame.** `_artemis_meta.json` is what distinguishes "a finished deploy" from "a pile of bytes someone abandoned". Everything in section 7 keys off its presence. + +Order matters and is not uniform across the codebase — see section 7.4. + +______________________________________________________________________ + +## 5. What is actually in the bucket + +The top level of the bucket **is** the site namespace. There is no `sites/` prefix. + +``` +test.freecode.camp/deploys/20260816-081716-sB68682/index.html +test.freecode.camp/deploys/20260816-081716-sB68682/_artemis_meta.json <- the marker +test.freecode.camp/production <- alias object +test.freecode.camp/preview <- alias object +_trash/test.freecode.camp/20260609-062751-it09864/index.html <- soft-deleted +``` + +An **alias object** is a tiny object whose *body* is a deploy id. That is the pointer. Caddy reads it to decide what to serve. This is why the architecture says **R2 is the only truth about what is live** — Postgres merely mirrors it. + +Three layout renderers exist and nothing cross-checks them at runtime: + +- `handler.DeployPrefixTemplate` — takes a **slug** (`internal/handler/deploykey.go`) +- `cmd/artemis/gcLayout` — takes a **dirname** (`cmd/artemis/gcwire.go:101`) +- the alias key formats — `ALIAS_PRODUCTION_KEY_FORMAT`, `ALIAS_PREVIEW_KEY_FORMAT` + +Nothing validates that they agree on where a site's tree begins. If they diverge, bytes and pointers land in different places and no test notices. + +`MovePrefix` (`internal/r2/r2.go:304`) is **not** a rename and **not** atomic: it is a per-object copy-then-delete loop that returns on first failure, leaving a deploy split between the live prefix and `_trash/`. + +______________________________________________________________________ + +## 6. The Postgres side + +| table | what it holds | +| ------------ | -------------------------------------------------------------------------- | +| `sites` | the registry: **slug**, authorized teams, creator | +| `deploys` | one row per indexed deploy, keyed `(site=dirname, id)`, with mtime + bytes | +| `aliases` | mirror of the R2 alias objects, keyed `(site=dirname, name)` | +| `tombstones` | one row per soft-deleted deploy — the purge worklist | +| `outbox` | events waiting to be published to the workflow engine | +| `audit_log` | append-only record of every privileged action | + +Note the keyspace split *inside the database*: `sites.slug` is a slug, every other `site` column is a dirname. + +**The site lock** is a Postgres advisory lock keyed on the site, taken by every mutation of that site — finalize, promote, rollback, delete, restore, purge, GC, and each reconcile repair. It is never explicitly released: the code opens a dedicated connection outside the pool and relies on closing it to drop the lock (`internal/pg/lock.go:15-34`). That works, but it means every locked request opens a new Postgres connection. + +With no Postgres configured the lock becomes a **silent no-op**, and concurrent alias writes race with no error. That is stated as a known limitation, not a bug. + +**The outbox** is the standard transactional-outbox pattern: a handler writes its state change and an event row in the same transaction, and a relay loop publishes the event afterwards. Nothing ever deletes published rows — the table grows forever. + +______________________________________________________________________ + +## 7. Cleanup: three different jobs people confuse constantly + +This is where most of the last 65 hours went, so it gets the most space. + +### 7.1 `gc-site` — retention + +Triggered by a `site.changed` event whenever a site's deploys change. Walks the **index** and retires deploys that are old and unreferenced: keeps anything an alias points at, keeps the N most recent, keeps anything inside the retention window (7 days by default, `internal/config/config.go:180`). Moves the rest to `_trash/` and writes a tombstone row. + +Because it walks the index, **a prefix with no index row is invisible to it, forever.** Hold that thought. + +### 7.2 `tombstone-purge` — the hard delete + +A nightly cron. Reads tombstone rows older than the recovery window (7 days), deletes the `_trash/` prefix for real, and clears the row (`internal/gc/tombstone.go:83-86`). This is the only path that destroys bytes irreversibly. + +### 7.3 `reconcile` — the drift repairer + +Compares R2 against Postgres for one site and classifies every disagreement into one of four **drift classes**. Learn these four; every report and alert is phrased in them. + +| class | what it means | danger | +| ------------------- | ------------------------------------------------------------------------ | -------- | +| **reindex** | complete, marked bytes in R2 with no index row | storage | +| **tombstone** | unmarked bytes past the grace window, no index row — an abandoned upload | storage | +| **prune** | an index row whose bytes are gone | **high** | +| **aliased-missing** | an alias points at a deploy that no longer exists | **high** | + +The first two are *reclaimable*: unreferenced bytes and forgotten rows. The last two mean something is already broken — `prune` deletes index rows, and `aliased-missing` means a live site serves nothing. + +Safety rails on the repair path: the site lock, a **second read inside the lock** before acting, a grace window so young deploys are never touched, and a blast cap. + +### 7.4 The ordering inconsistency you will trip over + +The two reaping paths write their two side effects in **opposite orders**: + +- `gc-site` moves the bytes, then records the tombstone row (`internal/gc/gcsite.go:117` then `:120`) +- `reconcile` records the row, then moves the bytes (`internal/gc/reconcile.go:275` then `:279`) + +Reconcile's order is the correct one: the purge is row-driven, so bytes moved without a row are a permanent leak. `gc-site` still has the leaky order — it is a known follow-up, not a fixed thing. + +______________________________________________________________________ + +## 8. How scheduled work runs + +There are no Kubernetes CronJobs for any of this. Workflows and crons are declared **in the Go binary** and registered with a Hatchet engine at boot (`cmd/artemis/gcworkflows.go:107`). Three workflows exist: + +| workflow | trigger | what it does | +| ----------------- | -------------------- | -------------------------- | +| `gc-site` | `site.changed` event | retention (7.1) | +| `tombstone-purge` | cron `0 3 * * *` | hard delete (7.2) | +| `drift-detect` | cron `0 4 * * *` | read-only sweep + alerting | + +An event travels: handler writes an `outbox` row in its transaction → the relay loop polls every 5s and publishes it to Hatchet → Hatchet starts the workflow. + +**Registration is additive and never subtractive.** The SDK only ever calls `PutWorkflow`; nothing deregisters a workflow the binary stopped declaring, and the engine's cron poller selects on `enabled` and the *version's* deleted flag — never on the workflow's. A retired cron therefore keeps firing forever with no worker to serve it, until someone disables it in the engine's database. That is operational knowledge you cannot recover from this repo alone. + +______________________________________________________________________ + +## 9. Why `drift-detect` is read-only *by type*, not by flag + +The old design had a `reconcile-scheduler` cron that repaired automatically. It was retired. The reasoning is worth understanding because it shapes the code: + +- The old cron enumerated **registry slugs** and handed them to a reconciler that expects **dirnames** (section 2). Under the production format those differ, so every nightly run looked at a prefix that does not exist, found zero drift, and exited successfully. It repaired nothing for months and never once alerted. +- When a read-only sweep finally measured production honestly, the two **dangerous** classes were empty and the two **storage** classes were not. + +So the repair capability stayed, but it is now started by a person. The cron that remains cannot repair — and crucially, not because a flag says so. `drift-detect` holds a reconciler whose store and mover are **read-only types** whose write methods return `errReadOnlyViolation`, and whose locker is `nil`. A repair from that job does not compile into a mutation. `TestGCWorkflowDefs_NoWorkflowCanRepairOnASchedule` pins it. + +The design principle: **a flag is a request; a type is a guarantee.** Every failure in this subsystem's history came from a job that *could* write and was merely asked not to. + +______________________________________________________________________ + +## 10. Where the bodies are buried + +Verified traps, each a real line of code. None of these are hypothetical. + +1. **`CLEANUP_BLAST_CAP` has no default.** It is absent from the defaults block (`internal/config/config.go:242-249`), so it is `0`, and both consumers treat `<= 0` as *disabled*. Production sets it to `10` explicitly; a fresh environment runs uncapped. +1. **A deploy's mtime is parsed out of its ID string**, not read from R2 metadata (`internal/gc/reconcile.go:494`). An ID whose first 15 characters are not `20060102-150405` gets a zero time. +1. **The marker extends a deploy's life, it does not shorten it.** A marked deploy is kept for the full retention window; an unmarked one only for the grace window. +1. **Reconcile records `bytes = 0`** on the tombstones it creates (`internal/gc/reconcile.go:275`), so purge's "bytes reclaimed" figure under-reports. +1. **Unknown `argv` silently boots the server.** `main.go:49` and `:56` compare against two exact strings with no default case and no usage text. `artemis --help` starts a web server. +1. **`driftreport` ignores its arguments** — `main.go:50` forwards nothing, unlike `main.go:57`. `artemis driftreport --site www` sweeps the whole fleet, silently. +1. **`BACKFILL_ON_BOOT` is a different program.** `runWith` does the backfill and returns before any listener starts, so the process exits 0 having served nothing. +1. **Bare `DELETE /api/site/{slug}` removes only the registry row.** Bytes, index rows and live alias objects all survive; the site just becomes unmanaged. `?purge=true` is the destructive one. +1. **There is no "already finalized" guard on upload.** A valid JWT can keep writing into a prefix that is already the live production target, for the rest of its TTL. +1. **`site-purge` writes a sentinel tombstone with `id = ''`**, and that row now blocks reindexing of *every* deploy in that site until the recovery window clears it. That is deliberate, added this week, and easy to mistake for a bug. +1. **Dead code that looks live.** `worker.RegisterDeployWorkflows` is never called outside tests — finalize, promote and rollback all run inline in the HTTP handlers. + +______________________________________________________________________ + +## 11. Reading order for the code itself + +1. `internal/handler/deploy.go` — the whole product in one file. +1. `internal/handler/deploykey.go` — 60 lines, and the source of every keyspace bug. +1. `internal/pg/repo.go` — every query, in one place. +1. `internal/gc/reconcile.go` — the hardest file; read `plan.go` first for the classes. +1. `cmd/artemis/main.go` + `gcwire.go` — how it is all assembled. + +To watch it work end to end without touching production: `go test ./cmd/artemis/ -run E2E` spins a real Postgres via testcontainers and a fake S3. diff --git a/docs/design/0005-drift-at-source.md b/docs/design/0005-drift-at-source.md new file mode 100644 index 0000000..f228c97 --- /dev/null +++ b/docs/design/0005-drift-at-source.md @@ -0,0 +1,187 @@ +# 0005 — Drift at source: fixing the cause, not the cleanup + +Status: proposed Supersedes the framing of [0004](0004-drift-detection-and-alerting.md) §rationale. + +## Why 0004 needs superseding + +0004 justified making the drift cron report-only on the grounds that repairable drift was "an edge case, not worth the risk of an automated repair loop". The first half of that sentence is wrong and the audit that produced this document proves it: + +| measure | value | source | +| ------------------------------------------ | -------------------- | ---------------------- | +| `deploy.init` audit rows, 33 days | 75 | `audit_log`, prod | +| `deploy.finalize` audit rows, same window | 55 | same | +| abandoned deploy sessions | 20 | difference | +| abandoned sessions that had uploaded bytes | 18 | drift report, per-site | +| all-time orphan prefixes in R2 | 32 of 37 drift items | `artemis driftreport` | +| `gc.reconcile` audit rows, all time | 2 | both mine, tonight | + +Roughly **18 orphan prefixes per month**, accumulating since the service was deployed, and nothing had ever collected them. + +The *decision* in 0004 stands — the cron stays read-only. The *reason* changes. Repair does not belong in a cron because repair should not be needed at all: an abandoned upload is not drift to be detected and reconciled after the fact, it is a deploy session the system never recorded. This document closes that hole, and in doing so changes what the cron's silence means — from "we chose not to look" to "there is genuinely nothing". + +## The root cause + +`POST /api/deploy/init` mints a JWT and returns. It writes an `audit_log` row (`internal/handler/deploy.go:90`) and nothing else. No row in `deploys` exists until `FinalizeAtomic` (`internal/pg/saga.go:11`) writes one at the *end* of a successful deploy. + +Between those two calls the client PUTs files into `/deploys//`. If the client then dies — CI cancelled, laptop closed, network dropped — those bytes exist in R2 and **no row anywhere names them**. Every reaper in the service walks the index: + +- `gc-site` plans from `DeploysForSite` (`internal/pg/repo.go:72`) → cannot see them. +- `tombstone-purge` walks `tombstones` → cannot see them. +- Only `reconcile`, which lists R2 directly and diffs against the index, can. + +That is the entire reason `reconcile` exists. It is a scanner built to find what a missing write should have recorded. + +## Design + +Four phases, in dependency order. P0 and P1 are independent of each other; P2 depends on neither but is the one that removes the drift class; P3 depends on P2. + +______________________________________________________________________ + +### P0 — Stop the bleeding + +Four small changes. Every one is a live defect today. + +#### P0-1. The production-format test harness (do this first) + +`DEPLOY_PREFIX_FORMAT` defaults to `/deploys/-/` (`internal/config/config.go:226`), which makes the registry slug and the storage dirname **identical strings**. Production uses `.freecode.camp/deploys/…`, where they differ. Every test in `internal/gc` and `cmd/artemis` runs under the default, so every keyspace confusion in this codebase is invisible to CI by construction — including both bugs below. + +Add a suite variant that runs the gc and CLI tests under an FQDN-shaped prefix format. This is the single highest-leverage change in the whole document: it is the harness that would have caught the original reconcile no-op *and* P0-2, and it is the harness that keeps P1 honest. + +Land it **RED first**, in the same commit as P0-2. + +#### P0-2. `LiveAliases` is inert in production (confirmed bug) + +`newLiveAliasReader` (`cmd/artemis/gcwire.go:133-157`) substitutes its `site` argument into the alias key format. Both call sites (`gcwire.go:190` for `SiteGC`, `gcwire.go:208` for `Reconciler`) pass a **storage dirname**, because that is what the sweep enumerates. The format expects a **slug**. In production that renders: + +``` +test.freecode.camp → test.freecode.camp.freecode.camp/production → always 404 +``` + +`r2.IsNotFound` is treated as "no alias" (`gcwire.go:150`), so the function returns an empty map for every site, silently. `LiveAliases` is the last-second re-read that stops gc from trashing a deploy an alias points at (`internal/gc/gcsite.go:107-112`) and the same guard in reconcile. **The safety net is currently inert in production.** It is not load-bearing today only because gc's plan already excludes aliased deploys from the index — but it was added precisely for the race where the plan is stale. + +The minimal principled fix, no slug plumbing required: + +`SiteDirname` is defined as the first path segment of the deploy prefix head (`internal/handler/deploykey.go:56-62`). The alias formats and the deploy prefix format share that head by construction in every real configuration — `.freecode.camp/production` and `.freecode.camp/deploys/…`. But **nothing validates it**. So: + +1. Validate at boot that each alias key format's first segment is byte-identical to the deploy prefix format's first segment. Refuse to start otherwise. +1. Given that invariant, the alias key for a dirname is exactly `dirname + "/" + tail`, where `tail` is the alias format after its first slash. No reverse lookup, no second keyspace conversion. + +The boot check is what makes step 2 correct rather than merely usually-correct, and it makes any future misconfiguration a startup failure instead of a silent 404. + +#### P0-3. `CLEANUP_BLAST_CAP` defaults to unlimited + +`config.go:545` reads the variable only `if ok` — there is no default, so `BlastCap` is `0`. Both consumers read `0` as *no cap*: + +```go +if rc.BlastCap <= 0 || destructive <= rc.BlastCap { return } // reconcile.go:213 +``` + +The env-var error message says "0 disables" (`config.go:548`), which is true but reads as "disables destruction" when it means "disables the limit on destruction". A safety valve whose default is off is not a safety valve. + +Change: default to a real number (10 is the natural choice — larger than any legitimate single-site cleanup, small enough to be a tripwire), and make `0` mean **refuse to perform destructive work**, not *unlimited*. Update the error string to match. + +#### P0-4. `gc-site` writes bytes before the row + +`gcsite.go` moves the prefix (`MovePrefix`, `internal/gc/gcsite.go:117`) and *then* records the tombstone (`Store.Tombstone`, `:120`). A crash between them leaves bytes in `_trash/` that no tombstone dates — invisible to `tombstone-purge`, so they are never hard-deleted, and invisible to the index. That is a third orphan class, created by the cleanup path itself. + +`reconcile` already does it in the safe order — `RecordTombstone` (`internal/gc/reconcile.go:275`) then `MovePrefix` (`:279`) — so a crash leaves a tombstone for bytes still in place, which is self-healing on retry. Make `gc-site` match. Row first, always. + +Note while touching this: `reconcile.go:275` hardcodes `bytes = 0` on the tombstone. Reclaimed-bytes accounting is therefore wrong for every reconcile repair. Low severity, fix in passing. + +______________________________________________________________________ + +### P1 — Make the bug class impossible + +P0-2 is the third bug in this class found in this codebase. Patching the third one does not stop the fourth. + +Introduce two distinct types — `Slug` and `Dirname` — so the compiler rejects the substitution that produced P0-2, and give the layout exactly one renderer. Today there are multiple key-rendering paths that are not cross-checked against each other; `DeployPrefixTemplate` is the authoritative one and the others should be deleted in its favour. + +Scope note: Postgres currently stores dirnames in `deploys.site`, while the registry and JWT carry slugs. Normalising the database to slugs is the *clean* end state but requires a migration and touches every query. **That is out of scope here.** The type split alone kills the class without any migration — `Dirname` is simply the type that crosses the R2 and Postgres boundaries, and `Slug` the type that crosses the HTTP and registry boundaries, with `SiteDirname` the single sanctioned conversion. + +______________________________________________________________________ + +### P2 — Remove the orphan source + +This is the answer to "what should we do about the GC". + +**Have `deploy.init` write a `state='pending'` row in `deploys`.** + +The seam is already there and unused. Verified: + +- `deploys.state` defaults to `'active'`; all 198 rows in production are `'active'`. +- Every read path filters `state = 'active'` (`internal/pg/repo.go:72,96`), so a `pending` row is invisible to gc planning, drift accounting and the API until it is promoted. +- Only `internal/backfill/backfill.go:96` and `cmd/loadgen/main.go:125` pass a state explicitly. +- `FinalizeAtomic` already upserts `ON CONFLICT (site, id) DO UPDATE SET … state = 'active'` (`internal/pg/saga.go:11-25`). **The promotion needs no code change at all** — finalize flips pending→active as a side effect of what it already does. `ReindexDeploy` does the same for the marker-written-but-crashed case. + +Then teach `gc-site` one rule: a `pending` row older than the grace window is an abandoned deploy — trash its prefix and record its tombstone, same as any retention deletion. The abandoned upload stops being an anomaly that requires a special scanner and becomes an ordinary retention case handled by the event-driven job that already works. + +Details that must be in the implementation, not discovered during it: + +- **The pending row must be written in the same keyspace finalize uses.** `deploy.go:285` passes `h.DeployPrefix.SiteDirname(claims.Site)` to `FinalizeAtomic` — a **dirname**. Init holds `claims.Site`, a **slug**. If the pending write uses the slug, `ON CONFLICT (site, id)` never matches: the row never flips to active, and at grace+72h `gc-site` "expires" a deploy that finalized perfectly well, trashing the prefix the *live* site is serving. This is P0-2's bug class reappearing inside its own fix. The pending→active promotion must be exercised under the P0-1 FQDN harness, where slug and dirname differ; under the default format the mistake is invisible. + +- **Failure mode of the init write.** Follow the existing audit precedent exactly (`internal/handler/handler.go:198-215`): nil-check the store, detached 5s timeout, log + Sentry on failure, **never fail the request**. Deploy-only mode with no index must keep working, and making Postgres a hard dependency of every `deploy/init` trades a rare storage leak for a total outage. Reconcile remains the backstop for the leak that survives this. + +- **Atomicity of expiry.** The pending row must be retired in the same transaction as the tombstone insert, or the next sweep counts the same deploy twice. + +- **The no-bytes path.** 2 of the 20 abandoned sessions in the audit window uploaded nothing. Expiring those must drop the row and skip the tombstone — a tombstone for a prefix that never existed is drift the sweep will then report forever. + +- **Threshold.** Grace must sit far above the deploy JWT TTL (15 minutes), so no in-flight deploy is ever reapable. 72h is the obvious choice and matches existing retention language. + +Effect: the tombstone/orphan drift class — 32 of the 37 items in the current report — stops being generated at all. + +______________________________________________________________________ + +### P3 — Then, and only then, re-tune the alerting + +This is the answer to "what should we do about the CRON". + +**Keep `drift-detect` exactly as shipped in 1.7.0.** Read-only by type (`readOnlyStore` / `readOnlyMover` / `Locker=nil`), report-only, nightly, Sentry check-in. That decision was right and nothing here changes it. What changes is what its silence *means*: today a clean report means "we deliberately do not collect the ~18/month we know accrues"; after P2 it means "there is genuinely nothing". + +That in turn simplifies the alert policy. The growth-derivative threshold deferred in 0004 becomes unnecessary — once the baseline is zero by construction, a small static threshold is real signal: + +- `aliased-missing > 0` → page. (unchanged, this is already a hard failure) +- self-check mismatch or unreadable sites → page. (unchanged) +- `reindex + tombstone > N` for small N → alert, because after P2 it means a new leak class exists that P2 does not cover. + +Also: the report should name deploy IDs and ages, not just counts. An operator reading a nightly alert needs to know *which* deploys and *how old* to decide whether it is one bad CI run or a systemic regression. + +**Reconcile stays human-run.** After P2 its remaining job is the genuine break-glass case: bytes in R2 with no `deploy.init` record at all — written out of band, or surviving a Postgres restore. That is rare, dangerous, and should require a human at the keyboard. + +#### The gap between now and P2 + +P0–P2 will take time to ship, and roughly 18 orphans per month accrue while the alert policy stays deliberately silent about them. Two honest options: + +1. Add the static `reclaimable > 50` branch now — one branch in `classifyDrift` and one test — so the accrual is at least visible. +1. Accept the window explicitly and rely on the manual `artemis reconcile` run. + +Recommendation: **(1)**. It is genuinely one branch, and an unwatched accrual with a known rate is exactly what monitoring is for. + +______________________________________________________________________ + +## Bug disposition + +Everything verified during this audit, with a decision against each. "Accept, documented" is a real disposition; silence is not. + +| # | Finding | Disposition | +| --- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| 1 | `LiveAliases` receives a dirname, alias format expects a slug → always 404, safety net inert in prod | **P0-2**, fix now | +| 2 | Whole gc/CLI suite runs under a format where slug == dirname, hiding the entire bug class | **P0-1**, fix now | +| 3 | `CLEANUP_BLAST_CAP` has no default → `0` → unlimited destruction | **P0-3**, fix now | +| 4 | `gc-site` moves bytes before writing the tombstone row | **P0-4**, fix now | +| 5 | `reconcile` records tombstones with hardcoded `bytes = 0` | **P0-4**, in passing | +| 6 | Abandoned `deploy.init` sessions leave unowned bytes (~18/month) | **P2**, the root cause | +| 7 | Two keyspaces with no type separation | **P1** | +| 8 | Postgres stores dirnames, registry stores slugs | **Out of scope** — migration; P1's types make it survivable | +| 9 | `runDriftReport` is called with no argv (`cmd/artemis/main.go:49`) so `driftreport ` silently ignores it; and any unrecognised subcommand falls through to `run()` (`:62`), i.e. **a mistyped subcommand starts the server** | **P0-adjacent** — operators started running these subcommands against production *this week*. A typo that boots a server, and a report that ignores the argument an operator typed, are both how a run gets misread as authoritative. Fix with P0: reject unknown subcommands, reject unexpected args. | +| 10 | A finalized deploy remains writable for the JWT's remaining TTL (up to 15 min) | **Accept, documented.** Real, but requires an authorized token holder; tightening it means invalidating the JWT at finalize, which is its own design. Record in ONBOARDING traps. | +| 11 | `outbox` has no retention — unbounded growth | **Backlog.** Small table, slow growth, no correctness impact. Needs a purge job eventually; not part of this wave. | +| 12 | Dead worker code paths | **Backlog**, cosmetic. | +| 13 | `RequireScope` / latched rate limiter behaviours | **Accept, documented.** Both behave as designed; the surprise is documentation, not code. Already captured in ONBOARDING §10. | + +## Sequencing + +P0 is one commit-sized wave and should ship on its own — four defects, all small, all independently testable, with P0-1 landed RED first so the P0-2 fix has a failing test to turn green. + +P1 and P2 are each their own wave. P3 is a follow-up to P2, except for the interim `reclaimable > 50` branch, which rides with P0. + +This document is the seed artifact for a new dossier. It does not belong in `artemis-audit-fixes`, which is at 19/20 with a pending converge. From 020e7d0a8c09be204ee8e0a43aa7d4892d558ad5 Mon Sep 17 00:00:00 2001 From: Mrugesh Mohapatra Date: Mon, 17 Aug 2026 09:36:33 +0530 Subject: [PATCH 02/41] docs: ground the domain layout and correct blast cap --- docs/design/0005-drift-at-source.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/docs/design/0005-drift-at-source.md b/docs/design/0005-drift-at-source.md index f228c97..1839275 100644 --- a/docs/design/0005-drift-at-source.md +++ b/docs/design/0005-drift-at-source.md @@ -31,6 +31,23 @@ Between those two calls the client PUTs files into `/deploys// That is the entire reason `reconcile` exists. It is a scanner built to find what a missing write should have recorded. +## Not a problem: the domain layout + +Raised during review, and worth recording so it is not re-litigated. The doubled string `test.freecode.camp.freecode.camp` **is not a hostname**. It is the malformed R2 object-key prefix that the P0-2 bug renders. Nothing resolves it, nothing requests a certificate for it; it 404s inside R2 and that is the whole of its blast radius. + +The real subdomain layout is coherent and verified: + +| host | R2 alias key | labels under root | +| ---------------------------- | ------------------------------- | ----------------- | +| `test.freecode.camp` | `test.freecode.camp/production` | 1 | +| `test.preview.freecode.camp` | `test.freecode.camp/preview` | 2 | + +The edge derives the key from the Host header — `parseSiteAndAlias` (`infra-backup/docker/images/caddy-s3/modules/r2alias/host.go:17-40`) strips the root domain, and if the last remaining label is the preview subdomain it returns `site = .`, `alias = "preview"`. So both hosts collapse to the *same* single-label site dirname and differ only by alias name. That is why `ALIAS_*_KEY_FORMAT` carry the FQDN (`values.production.yaml:53-55`) — the key must match what the edge computes. + +Two-label preview hosts are covered by TLS: the live certificate SAN is `*.freecode.camp, *.preview.freecode.camp, freecode.camp` (probed 2026-08-17 against both hosts). A single `*.freecode.camp` wildcard would **not** match `test.preview.freecode.camp` — RFC 6125 wildcards span exactly one label — but a second explicit wildcard is present, so preview is correctly served. This is load-bearing: production holds **70 preview aliases across 70 sites** versus 57 production aliases, most recent 2026-08-16. + +No change proposed here. + ## Design Four phases, in dependency order. P0 and P1 are independent of each other; P2 depends on neither but is the one that removes the drift class; P3 depends on P2. @@ -70,6 +87,8 @@ The boot check is what makes step 2 correct rather than merely usually-correct, #### P0-3. `CLEANUP_BLAST_CAP` defaults to unlimited +**Severity corrected after probing infra: production is _not_ at risk.** `values.production.yaml:73` sets `CLEANUP_BLAST_CAP: "10"`, so the live service is capped. The defect is that both fallbacks are `0` = uncapped — the chart default (`charts/artemis/values.yaml:141`, commented "0 = uncapped") and the code default below. Any new environment, or a values file that forgets the override, runs destructive cleanup with no ceiling. A footgun, not a live fire. + `config.go:545` reads the variable only `if ok` — there is no default, so `BlastCap` is `0`. Both consumers read `0` as *no cap*: ```go @@ -177,6 +196,7 @@ Everything verified during this audit, with a decision against each. "Accept, do | 11 | `outbox` has no retention — unbounded growth | **Backlog.** Small table, slow growth, no correctness impact. Needs a purge job eventually; not part of this wave. | | 12 | Dead worker code paths | **Backlog**, cosmetic. | | 13 | `RequireScope` / latched rate limiter behaviours | **Accept, documented.** Both behave as designed; the surprise is documentation, not code. Already captured in ONBOARDING §10. | +| 14 | `PublicURLForSite` (`internal/handler/handler.go:152`) is never assigned outside tests, so the hardcoded fallback at `deploy.go:377-382` always runs — the public URL returned to the CLI bakes `freecode.camp` and `.preview.` into the binary, while every other domain fact comes from config. There is no `ROOT_DOMAIN` setting. | **Fix with P0** (one-liner): derive the URL from the configured alias formats, or add the root domain to config. Cosmetic today, silently wrong the day the root domain or preview label changes. | ## Sequencing From 9fcad988d2bdb12cadd9d6d69b182bf894471eec Mon Sep 17 00:00:00 2001 From: Mrugesh Mohapatra Date: Mon, 17 Aug 2026 09:54:47 +0530 Subject: [PATCH 03/41] fix(gc): read live aliases in the sweep keyspace --- cmd/artemis/gcwire.go | 38 ++++++++++++++++++++++----- cmd/artemis/gcwire_test.go | 41 +++++++++++++++++++++++++++--- cmd/artemis/workflowerrors_test.go | 8 +++--- 3 files changed, 73 insertions(+), 14 deletions(-) diff --git a/cmd/artemis/gcwire.go b/cmd/artemis/gcwire.go index 9312f02..8e86a94 100644 --- a/cmd/artemis/gcwire.go +++ b/cmd/artemis/gcwire.go @@ -130,18 +130,41 @@ type aliasGetter interface { GetAlias(ctx context.Context, aliasKey string) (string, error) } -func newLiveAliasReader(getter aliasGetter, formats ...string) (func(context.Context, string) (map[string]struct{}, error), error) { - fmts := make([]string, 0, len(formats)) +func siteSegment(format string) (string, error) { + slash := strings.IndexByte(format, '/') + if slash < 0 { + return "", fmt.Errorf("key format %q must contain '/' after the site segment", format) + } + return format[:slash], nil +} + +func newLiveAliasReader(getter aliasGetter, deployFormat string, formats ...string) (func(context.Context, string) (map[string]struct{}, error), error) { + deploySeg, err := siteSegment(deployFormat) + if err != nil { + return nil, fmt.Errorf("DEPLOY_PREFIX_FORMAT: %w", err) + } + tails := make([]string, 0, len(formats)) for _, f := range formats { if !strings.Contains(f, "") { return nil, fmt.Errorf("alias key format %q must contain ", f) } - fmts = append(fmts, f) + seg, err := siteSegment(f) + if err != nil { + return nil, err + } + if seg != deploySeg { + return nil, fmt.Errorf( + "alias key format %q has site segment %q but DEPLOY_PREFIX_FORMAT %q has %q: "+ + "the GC sweep enumerates storage dirnames rendered from the deploy prefix, so an alias "+ + "key under a different site segment is unreachable and would 404 for every site", + f, seg, deployFormat, deploySeg) + } + tails = append(tails, f[len(seg)+1:]) } - return func(ctx context.Context, site string) (map[string]struct{}, error) { + return func(ctx context.Context, dirname string) (map[string]struct{}, error) { out := map[string]struct{}{} - for _, f := range fmts { - v, err := getter.GetAlias(ctx, strings.ReplaceAll(f, "", site)) + for _, tail := range tails { + v, err := getter.GetAlias(ctx, dirname+"/"+tail) if err != nil { if r2.IsNotFound(err) { continue @@ -177,7 +200,8 @@ func newGCWiring(cfg *config.Config, repo *pg.Repo, r2c *r2.Client) (*gcWiring, if err != nil { return nil, err } - liveAliases, err := newLiveAliasReader(r2c, cfg.Aliases.ProductionKeyFormat, cfg.Aliases.PreviewKeyFormat) + liveAliases, err := newLiveAliasReader(r2c, cfg.DeployPrefixFormat, + cfg.Aliases.ProductionKeyFormat, cfg.Aliases.PreviewKeyFormat) if err != nil { return nil, err } diff --git a/cmd/artemis/gcwire_test.go b/cmd/artemis/gcwire_test.go index d307d2a..542d540 100644 --- a/cmd/artemis/gcwire_test.go +++ b/cmd/artemis/gcwire_test.go @@ -3,10 +3,12 @@ package main import ( "context" "errors" + "strings" "testing" "time" "github.com/freeCodeCamp/artemis/internal/config" + "github.com/freeCodeCamp/artemis/internal/handler" "github.com/freeCodeCamp/artemis/internal/pg" "github.com/freeCodeCamp/artemis/internal/r2" "github.com/stretchr/testify/assert" @@ -89,23 +91,54 @@ func TestNewLiveAliasReader_KeyMatchesWritePath(t *testing.T) { getter := &recordingAliasGetter{values: map[string]string{ "www.freecode.camp/production": "20260101-000000-abc1234", }} - read, err := newLiveAliasReader(getter, prodFmt) + read, err := newLiveAliasReader(getter, domainFormat, prodFmt) require.NoError(t, err) - live, err := read(context.Background(), "www") + live, err := read(context.Background(), "www.freecode.camp") require.NoError(t, err) assert.Equal(t, []string{"www.freecode.camp/production"}, getter.keys, - "GC live re-read must query the write-path key (ReplaceAll ), not a slash-derived tail") + "both call sites pass the storage dirname the sweep enumerates, so substituting it into "+ + " renders www.freecode.camp.freecode.camp/production and 404s forever") _, ok := live["20260101-000000-abc1234"] assert.True(t, ok, "the live deploy behind the prod alias must be detected by the pre-delete safety net") } +func TestNewLiveAliasReader_ReadsTheSameKeyspaceTheSweepEnumerates(t *testing.T) { + tmpl, err := handler.NewDeployPrefixTemplate(domainFormat) + require.NoError(t, err) + layout, err := newGCLayout(domainFormat, "_trash/") + require.NoError(t, err) + + getter := &recordingAliasGetter{} + read, err := newLiveAliasReader(getter, domainFormat, + ".freecode.camp/production", ".freecode.camp/preview") + require.NoError(t, err) + + dirname := tmpl.SiteDirname("test") + _, err = read(context.Background(), dirname) + require.NoError(t, err) + + require.Len(t, getter.keys, 2) + for _, k := range getter.keys { + assert.True(t, strings.HasPrefix(k, layout.sitePrefix(dirname)[:len(dirname)+1]), + "alias key %q must sit under the same site directory the sweep lists, or the "+ + "pre-delete safety net reads a prefix no alias was ever written to", k) + } +} + func TestNewLiveAliasReader_RequiresSiteToken(t *testing.T) { - _, err := newLiveAliasReader(&recordingAliasGetter{}, "production/only") + _, err := newLiveAliasReader(&recordingAliasGetter{}, domainFormat, "production/only") require.Error(t, err, "an alias format missing must fail boot, not silently mis-derive keys") } +func TestNewLiveAliasReader_RejectsASiteSegmentTheDeployPrefixDoesNotShare(t *testing.T) { + _, err := newLiveAliasReader(&recordingAliasGetter{}, domainFormat, ".preview.freecode.camp/production") + require.Error(t, err, + "an alias format whose site segment differs from the deploy prefix's cannot be reached from a "+ + "dirname; boot must refuse rather than 404 silently for every site") +} + func TestOpenRepoQueue_RequiresDatabase(t *testing.T) { q, err := openRepoQueue(nil) require.Error(t, err, "repo feature without a database must be rejected at boot") diff --git a/cmd/artemis/workflowerrors_test.go b/cmd/artemis/workflowerrors_test.go index 6fc0399..59359d6 100644 --- a/cmd/artemis/workflowerrors_test.go +++ b/cmd/artemis/workflowerrors_test.go @@ -135,13 +135,15 @@ func (g stubAliasGetter) GetAlias(context.Context, string) (string, error) { return g.val, g.err } +const bareFormat = "/deploys/-/" + func TestNewLiveAliasReader_RejectsFormatWithoutSiteToken(t *testing.T) { - _, err := newLiveAliasReader(stubAliasGetter{}, "no-token/production") + _, err := newLiveAliasReader(stubAliasGetter{}, bareFormat, "no-token/production") require.ErrorContains(t, err, "") } func TestNewLiveAliasReader_PropagatesGetterError(t *testing.T) { - read, err := newLiveAliasReader(stubAliasGetter{err: errors.New("r2 down")}, "/production") + read, err := newLiveAliasReader(stubAliasGetter{err: errors.New("r2 down")}, bareFormat, "/production") require.NoError(t, err) _, err = read(context.Background(), "www") @@ -149,7 +151,7 @@ func TestNewLiveAliasReader_PropagatesGetterError(t *testing.T) { } func TestNewLiveAliasReader_CollectsTargets(t *testing.T) { - read, err := newLiveAliasReader(stubAliasGetter{val: "d1"}, "/production", "/preview") + read, err := newLiveAliasReader(stubAliasGetter{val: "d1"}, bareFormat, "/production", "/preview") require.NoError(t, err) got, err := read(context.Background(), "www") From 94acceda378e33579f4868f81f67f9dd0b5966cd Mon Sep 17 00:00:00 2001 From: Mrugesh Mohapatra Date: Mon, 17 Aug 2026 09:54:54 +0530 Subject: [PATCH 04/41] fix(cli): reject unknown subcommands and stray args --- cmd/artemis/dispatch_test.go | 37 ++++++++++++++++++ cmd/artemis/main.go | 73 ++++++++++++++++++++++-------------- 2 files changed, 82 insertions(+), 28 deletions(-) create mode 100644 cmd/artemis/dispatch_test.go diff --git a/cmd/artemis/dispatch_test.go b/cmd/artemis/dispatch_test.go new file mode 100644 index 0000000..f159e0c --- /dev/null +++ b/cmd/artemis/dispatch_test.go @@ -0,0 +1,37 @@ +package main + +import ( + "bytes" + "context" + "testing" + + "github.com/stretchr/testify/require" +) + +func TestDispatchSubcommand_NoArgsRunsTheServer(t *testing.T) { + t.Parallel() + + handled, err := dispatchSubcommand(context.Background(), &bytes.Buffer{}, nil) + require.NoError(t, err) + require.False(t, handled, "an argv-less invocation is the server, not a subcommand") +} + +func TestDispatchSubcommand_RejectsAnUnknownSubcommand(t *testing.T) { + t.Parallel() + + handled, err := dispatchSubcommand(context.Background(), &bytes.Buffer{}, []string{"drift-report"}) + require.True(t, handled, + "an unrecognised subcommand must not fall through to the server: a typo would boot a second "+ + "artemis instead of reporting the typo") + require.ErrorContains(t, err, "drift-report") +} + +func TestDispatchSubcommand_RejectsArgumentsToDriftReport(t *testing.T) { + t.Parallel() + + handled, err := dispatchSubcommand(context.Background(), &bytes.Buffer{}, []string{driftReportCommand, "www"}) + require.True(t, handled) + require.ErrorContains(t, err, "takes no arguments", + "driftreport swept every site regardless of argv, so a site name an operator typed was silently "+ + "ignored and the whole-fleet report read as scoped") +} diff --git a/cmd/artemis/main.go b/cmd/artemis/main.go index a45f896..f4d114f 100644 --- a/cmd/artemis/main.go +++ b/cmd/artemis/main.go @@ -10,11 +10,13 @@ import ( "context" "errors" "fmt" + "io" "log/slog" "net/http" "os" "os/signal" "strconv" + "strings" "syscall" "time" @@ -45,17 +47,29 @@ var ( const bootPhaseTimeout = 20 * time.Second -func main() { - if len(os.Args) > 1 && os.Args[1] == driftReportCommand { - if err := runDriftReport(context.Background(), os.Stdout); err != nil { - fmt.Fprintln(os.Stderr, "drift report failed:", err) - os.Exit(1) +func dispatchSubcommand(ctx context.Context, out io.Writer, args []string) (bool, error) { + if len(args) == 0 { + return false, nil + } + switch args[0] { + case driftReportCommand: + if len(args) > 1 { + return true, fmt.Errorf("%s takes no arguments, got %q: it always sweeps every registered site", + driftReportCommand, strings.Join(args[1:], " ")) } - return + return true, runDriftReport(ctx, out) + case reconcileCommand: + return true, runReconcileCLI(ctx, out, args[1:]) + default: + return true, fmt.Errorf("unknown subcommand %q: expected %s or %s", + args[0], driftReportCommand, reconcileCommand) } - if len(os.Args) > 1 && os.Args[1] == reconcileCommand { - if err := runReconcileCLI(context.Background(), os.Stdout, os.Args[2:]); err != nil { - fmt.Fprintln(os.Stderr, "reconcile failed:", err) +} + +func main() { + if handled, err := dispatchSubcommand(context.Background(), os.Stdout, os.Args[1:]); handled { + if err != nil { + fmt.Fprintln(os.Stderr, "artemis:", err) os.Exit(1) } return @@ -450,25 +464,28 @@ type handlerDeps struct { func buildHandlers(cfg *config.Config, d handlerDeps) *handler.Handlers { h := &handler.Handlers{ - GH: d.gh, - JWT: d.jwt, - Sites: d.sites, - Registry: d.registry, - Health: d.health, - R2: d.r2, - AliasProductionFmt: cfg.Aliases.ProductionKeyFormat, - AliasPreviewFmt: cfg.Aliases.PreviewKeyFormat, - DeployPrefix: d.deployPrefix, - TrashPrefixBase: cfg.Cleanup.TrashPrefix, - TrashRecovery: time.Duration(cfg.Cleanup.RecoveryDays) * 24 * time.Hour, - UploadMaxBytes: cfg.UploadMaxBytes, - RegistryAuthzTeam: cfg.Registry.AuthzTeam, - RepoOrg: cfg.Repo.Org, - RepoCreateAuthzTeam: cfg.Repo.CreateAuthzTeam, - RepoApproveAuthzTeam: cfg.Repo.ApproveAuthzTeam, - AuditReadAuthzTeam: cfg.Repo.AuditReadAuthzTeam, - NewDeployID: r2.NewDeployID, - Now: time.Now, + GH: d.gh, + JWT: d.jwt, + Sites: d.sites, + Registry: d.registry, + Health: d.health, + R2: d.r2, + AliasProductionFmt: cfg.Aliases.ProductionKeyFormat, + AliasPreviewFmt: cfg.Aliases.PreviewKeyFormat, + + PublicProductionURLFmt: cfg.Aliases.ProductionURLFormat, + PublicPreviewURLFmt: cfg.Aliases.PreviewURLFormat, + DeployPrefix: d.deployPrefix, + TrashPrefixBase: cfg.Cleanup.TrashPrefix, + TrashRecovery: time.Duration(cfg.Cleanup.RecoveryDays) * 24 * time.Hour, + UploadMaxBytes: cfg.UploadMaxBytes, + RegistryAuthzTeam: cfg.Registry.AuthzTeam, + RepoOrg: cfg.Repo.Org, + RepoCreateAuthzTeam: cfg.Repo.CreateAuthzTeam, + RepoApproveAuthzTeam: cfg.Repo.ApproveAuthzTeam, + AuditReadAuthzTeam: cfg.Repo.AuditReadAuthzTeam, + NewDeployID: r2.NewDeployID, + Now: time.Now, } h.RepoGH = d.repoGH if cfg.Repo.Enabled() { From a5ed663b73c1d7e825e64f5249079909c3147cb5 Mon Sep 17 00:00:00 2001 From: Mrugesh Mohapatra Date: Mon, 17 Aug 2026 09:54:54 +0530 Subject: [PATCH 05/41] fix(gc): record the tombstone row before moving bytes --- internal/gc/errorpath_test.go | 12 ++++-- internal/gc/gcsite.go | 10 +++-- internal/gc/gcsite_ordering_test.go | 64 +++++++++++++++++++++++++++++ internal/gc/reconcile.go | 15 ++++++- 4 files changed, 92 insertions(+), 9 deletions(-) create mode 100644 internal/gc/gcsite_ordering_test.go diff --git a/internal/gc/errorpath_test.go b/internal/gc/errorpath_test.go index ff5e299..8262dd9 100644 --- a/internal/gc/errorpath_test.go +++ b/internal/gc/errorpath_test.go @@ -108,11 +108,13 @@ func TestGC_TombstoneRecordFailurePropagates(t *testing.T) { require.ErrorContains(t, err, "record tombstone") assert.Empty(t, res.Tombstoned, "a failed PG tombstone is not reported as reclaimed") assert.EqualValues(t, 0, res.BytesReclaimed, "no bytes accounted for an unrecorded tombstone") - require.Len(t, mover.moves, 1, "the R2 move ran before the PG write failed, leaving orphaned bytes the retry must reclaim") + assert.Empty(t, mover.moves, + "the row dates the bytes, so it lands first; moving bytes whose row is known to have failed strands "+ + "them in _trash/ where neither tombstone-purge nor reconcile lists them") assert.Empty(t, store.tombstoned) } -func TestGC_MoveFailureAbortsBeforeTombstone(t *testing.T) { +func TestGC_MoveFailureLeavesTheTombstoneRowForTheNextRun(t *testing.T) { mover := &errMover{err: errors.New("r2 5xx")} store := &fakeStore{ deploys: map[string][]Deploy{"www": sixOld()}, @@ -122,8 +124,10 @@ func TestGC_MoveFailureAbortsBeforeTombstone(t *testing.T) { res, err := newSiteGC(store, mover).Run(context.Background(), "www", false) require.ErrorContains(t, err, "tombstone-move") - assert.Empty(t, store.tombstoned, "no PG tombstone when the R2 move failed (V1/V5)") - assert.Empty(t, res.Tombstoned) + assert.Equal(t, []string{"www/d-old"}, store.tombstoned, + "the row landed before the move, so the bytes stay at the deploy prefix and surface as reindex "+ + "drift — visible to drift-detect and repairable by reconcile, unlike bytes stranded in _trash/") + assert.Empty(t, res.Tombstoned, "a deploy whose bytes never moved is not reported as reclaimed") require.Len(t, mover.moves, 1, "aborts on the first failed move, never proceeding to the next deploy") } diff --git a/internal/gc/gcsite.go b/internal/gc/gcsite.go index 0984258..bc8c2ea 100644 --- a/internal/gc/gcsite.go +++ b/internal/gc/gcsite.go @@ -114,12 +114,16 @@ func (g *SiteGC) Run(ctx context.Context, site string, dryRun bool) (GCResult, e } src := g.DeployPrefix(site, d.ID) dst := g.TrashPrefix(site, d.ID) - if _, err := g.Mover.MovePrefix(opCtx, src, dst); err != nil { - return fmt.Errorf("tombstone-move %s: %w", d.ID, err) - } if err := g.Store.Tombstone(opCtx, site, d); err != nil { return fmt.Errorf("record tombstone %s: %w", d.ID, err) } + if _, err := g.Mover.MovePrefix(opCtx, src, dst); err != nil { + slog.WarnContext(opCtx, "gc.site.tombstone_move_deferred", + "site", site, "deploy_id", d.ID, "trash_prefix", dst, "err", err, + "detail", "the row landed before the move, so the bytes stay at the deploy prefix and "+ + "surface as reindex drift for the next drift sweep") + return fmt.Errorf("tombstone-move %s: %w", d.ID, err) + } res.Tombstoned = append(res.Tombstoned, d.ID) res.BytesReclaimed += d.Bytes tombstoned = true diff --git a/internal/gc/gcsite_ordering_test.go b/internal/gc/gcsite_ordering_test.go new file mode 100644 index 0000000..bcaebec --- /dev/null +++ b/internal/gc/gcsite_ordering_test.go @@ -0,0 +1,64 @@ +package gc + +import ( + "context" + "errors" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +type orderRecorder struct { + *fakeStore + mover *fakeMover + order []string +} + +func (r *orderRecorder) Tombstone(ctx context.Context, site string, d Deploy) error { + r.order = append(r.order, "row") + return r.fakeStore.Tombstone(ctx, site, d) +} + +func (r *orderRecorder) MovePrefix(ctx context.Context, src, dst string) (int, error) { + r.order = append(r.order, "bytes") + return r.mover.MovePrefix(ctx, src, dst) +} + +func TestSiteGC_RecordsTheTombstoneRowBeforeMovingBytes(t *testing.T) { + rec := &orderRecorder{ + fakeStore: &fakeStore{deploys: map[string][]Deploy{"www": sixOld()}}, + mover: &fakeMover{}, + } + g := newSiteGC(rec, rec) + + res, err := g.Run(context.Background(), "www", false) + require.NoError(t, err) + require.NotEmpty(t, res.Tombstoned) + + require.NotEmpty(t, rec.order) + assert.Equal(t, "row", rec.order[0], + "a crash between the two writes must leave a tombstone for bytes still in place (self-healing on "+ + "retry), never bytes in _trash/ that no tombstone dates — those are invisible to tombstone-purge "+ + "and to the index, so they are never hard-deleted") +} + +type rowFailStore struct { + *fakeStore +} + +func (s *rowFailStore) Tombstone(context.Context, string, Deploy) error { + return errors.New("pg down") +} + +func TestSiteGC_LeavesBytesInPlaceWhenTheTombstoneRowFails(t *testing.T) { + store := &rowFailStore{fakeStore: &fakeStore{deploys: map[string][]Deploy{"www": sixOld()}}} + mover := &fakeMover{} + g := newSiteGC(store, mover) + + _, err := g.Run(context.Background(), "www", false) + require.Error(t, err) + + assert.Empty(t, mover.moves, + "bytes must not move once the row that would date them is known to have failed") +} diff --git a/internal/gc/reconcile.go b/internal/gc/reconcile.go index d036a9c..d62eff1 100644 --- a/internal/gc/reconcile.go +++ b/internal/gc/reconcile.go @@ -50,6 +50,8 @@ type DriftReport struct { CapReason string } +const orphanBytesUnknown int64 = 0 + type r2Deploy struct { hasMarker bool mtime time.Time @@ -210,7 +212,16 @@ func (rc *Reconciler) classify(ctx context.Context, site string, snap siteSnapsh func (rc *Reconciler) applyBlastCap(ctx context.Context, site string, plan *repairPlan, report *DriftReport) { destructive := len(plan.tombstone) + len(plan.prune) - if rc.BlastCap <= 0 || destructive <= rc.BlastCap { + if destructive == 0 || (rc.BlastCap > 0 && destructive <= rc.BlastCap) { + return + } + if rc.BlastCap <= 0 { + report.Capped = true + report.CapReason = fmt.Sprintf( + "refusing %d destructive repairs: blast-cap 0 means no ceiling was configured", destructive) + plan.tombstone = nil + plan.prune = nil + slog.WarnContext(ctx, "reconcile.capped", "site", site, "reason", report.CapReason) return } report.Capped = true @@ -272,7 +283,7 @@ func (rc *Reconciler) repair(ctx context.Context, sess LockSession, site string, return false, err } trash := rc.TrashPrefix(site, id) - if err := rc.Store.RecordTombstone(opCtx, site, id, 0); err != nil { + if err := rc.Store.RecordTombstone(opCtx, site, id, orphanBytesUnknown); err != nil { return false, fmt.Errorf("record orphan %s: %w", id, err) } rowRecorded = true From 5b9bb0fb29b767f46241156040483a30d7c0aac6 Mon Sep 17 00:00:00 2001 From: Mrugesh Mohapatra Date: Mon, 17 Aug 2026 09:54:54 +0530 Subject: [PATCH 06/41] fix(gc): make a zero blast cap refuse, not unleash --- internal/config/config.go | 23 ++++++++++++++++++- internal/config/config_test.go | 2 +- internal/gc/blastcap_test.go | 42 ++++++++++++++++++++++++++++++++++ internal/gc/reconcile_test.go | 1 + 4 files changed, 66 insertions(+), 2 deletions(-) create mode 100644 internal/gc/blastcap_test.go diff --git a/internal/config/config.go b/internal/config/config.go index f4f9c90..05a4379 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -112,6 +112,8 @@ type JWTConfig struct { type AliasConfig struct { ProductionKeyFormat string PreviewKeyFormat string + ProductionURLFormat string + PreviewURLFormat string } // RegistryConfig holds the Valkey-backed registry settings: connection @@ -179,6 +181,7 @@ const ( serveCacheTTL = 15 * time.Second defaultCleanupRetentionDays = 7 defaultCleanupRecentKeep = 3 + defaultCleanupBlastCap = 10 defaultCleanupGrace = 72 * time.Hour defaultCleanupRecoveryDays = 7 defaultCleanupTrashPrefix = "_trash/" @@ -222,6 +225,8 @@ func Load() (*Config, error) { Aliases: AliasConfig{ ProductionKeyFormat: "/production", PreviewKeyFormat: "/preview", + ProductionURLFormat: "https://.freecode.camp", + PreviewURLFormat: "https://.preview.freecode.camp", }, DeployPrefixFormat: "/deploys/-/", UploadMaxBytes: 100 * 1024 * 1024, // 100 MiB @@ -242,6 +247,7 @@ func Load() (*Config, error) { Cleanup: CleanupConfig{ RetentionDays: defaultCleanupRetentionDays, RecentKeep: defaultCleanupRecentKeep, + BlastCap: defaultCleanupBlastCap, Grace: defaultCleanupGrace, TrashPrefix: defaultCleanupTrashPrefix, RecoveryDays: defaultCleanupRecoveryDays, @@ -294,6 +300,12 @@ func Load() (*Config, error) { if v, ok := os.LookupEnv("ALIAS_PREVIEW_KEY_FORMAT"); ok && v != "" { cfg.Aliases.PreviewKeyFormat = v } + if v, ok := os.LookupEnv("PUBLIC_URL_PRODUCTION_FORMAT"); ok && v != "" { + cfg.Aliases.ProductionURLFormat = v + } + if v, ok := os.LookupEnv("PUBLIC_URL_PREVIEW_FORMAT"); ok && v != "" { + cfg.Aliases.PreviewURLFormat = v + } if v, ok := os.LookupEnv("DEPLOY_PREFIX_FORMAT"); ok && v != "" { cfg.DeployPrefixFormat = v } @@ -420,6 +432,14 @@ func (c *Config) validate() error { if err := validateDeployPrefixFormat(c.DeployPrefixFormat); err != nil { return err } + for env, f := range map[string]string{ + "PUBLIC_URL_PRODUCTION_FORMAT": c.Aliases.ProductionURLFormat, + "PUBLIC_URL_PREVIEW_FORMAT": c.Aliases.PreviewURLFormat, + } { + if !strings.Contains(f, "") { + return fmt.Errorf("invalid %s %q: must contain ", env, f) + } + } if err := validateGitHubAPIBase(c.GitHub.APIBase); err != nil { return err } @@ -545,7 +565,8 @@ func loadCleanup(c *CleanupConfig) error { if v, ok := os.LookupEnv("CLEANUP_BLAST_CAP"); ok { n, err := strconv.Atoi(v) if err != nil || n < 0 { - return fmt.Errorf("invalid CLEANUP_BLAST_CAP %q: must be non-negative integer (0 disables)", v) + return fmt.Errorf("invalid CLEANUP_BLAST_CAP %q: must be non-negative integer "+ + "(0 refuses every destructive repair)", v) } c.BlastCap = n } diff --git a/internal/config/config_test.go b/internal/config/config_test.go index a4c96bf..c2ec244 100644 --- a/internal/config/config_test.go +++ b/internal/config/config_test.go @@ -210,7 +210,7 @@ func TestConfigLoad(t *testing.T) { assert.Equal(t, 7, cfg.Cleanup.RetentionDays) assert.Equal(t, 3, cfg.Cleanup.RecentKeep) assert.Equal(t, 72*time.Hour, cfg.Cleanup.Grace) - assert.Equal(t, 0, cfg.Cleanup.BlastCap) + assert.Equal(t, 10, cfg.Cleanup.BlastCap) assert.Equal(t, "_trash/", cfg.Cleanup.TrashPrefix) assert.Equal(t, 7, cfg.Cleanup.RecoveryDays) assert.False(t, cfg.Cleanup.DryRun) diff --git a/internal/gc/blastcap_test.go b/internal/gc/blastcap_test.go new file mode 100644 index 0000000..437b690 --- /dev/null +++ b/internal/gc/blastcap_test.go @@ -0,0 +1,42 @@ +package gc + +import ( + "context" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +const defaultTestBlastCap = 1000 + +func TestApplyBlastCap_ZeroRefusesEveryDestructiveRepair(t *testing.T) { + t.Parallel() + + rc := &Reconciler{BlastCap: 0} + plan := &repairPlan{tombstone: []string{"a", "b"}, prune: []string{"c"}} + report := &DriftReport{} + + rc.applyBlastCap(context.Background(), "www", plan, report) + + assert.Empty(t, plan.tombstone, + "an unset blast cap once meant unlimited, so a misconfigured environment reaped a whole site in "+ + "one run; zero must refuse instead") + assert.Empty(t, plan.prune) + require.True(t, report.Capped) + assert.Contains(t, report.CapReason, "blast-cap 0") +} + +func TestApplyBlastCap_LeavesAPlanWithinTheCapAlone(t *testing.T) { + t.Parallel() + + rc := &Reconciler{BlastCap: 10} + plan := &repairPlan{tombstone: []string{"a", "b"}, prune: []string{"c"}} + report := &DriftReport{} + + rc.applyBlastCap(context.Background(), "www", plan, report) + + assert.Len(t, plan.tombstone, 2) + assert.Len(t, plan.prune, 1) + assert.False(t, report.Capped) +} diff --git a/internal/gc/reconcile_test.go b/internal/gc/reconcile_test.go index a696746..6237980 100644 --- a/internal/gc/reconcile_test.go +++ b/internal/gc/reconcile_test.go @@ -98,6 +98,7 @@ func newReconciler(lister ReconcileLister, store ReconcileStore, mover Mover) *R Store: store, Mover: mover, Grace: time.Hour, + BlastCap: defaultTestBlastCap, SitePrefix: func(site string) string { return site + "/deploys/" }, DeployPrefix: func(site, id string) string { return site + "/deploys/" + id + "/" }, TrashPrefix: func(site, id string) string { return "_trash/" + site + "/" + id + "/" }, From 11eb897e4641a563e5d969d218c1716052c7d415 Mon Sep 17 00:00:00 2001 From: Mrugesh Mohapatra Date: Mon, 17 Aug 2026 09:54:55 +0530 Subject: [PATCH 07/41] fix(handler): build public URLs from config --- internal/config/publicurl_test.go | 65 +++++++++++++++++++++++++++ internal/handler/deploy.go | 7 +-- internal/handler/handler.go | 20 +++++---- internal/handler/publicurl_test.go | 20 +++++++++ internal/handler/test_helpers_test.go | 10 ++--- 5 files changed, 101 insertions(+), 21 deletions(-) create mode 100644 internal/config/publicurl_test.go create mode 100644 internal/handler/publicurl_test.go diff --git a/internal/config/publicurl_test.go b/internal/config/publicurl_test.go new file mode 100644 index 0000000..5a0b2b2 --- /dev/null +++ b/internal/config/publicurl_test.go @@ -0,0 +1,65 @@ +package config + +import ( + "testing" + + "github.com/stretchr/testify/require" +) + +func TestLoad_PublicURLFormatsDefaultToTheServedHostShapes(t *testing.T) { + t.Setenv("GH_CLIENT_ID", "cid") + t.Setenv("JWT_SIGNING_KEY", "0123456789abcdef0123456789abcdef") + t.Setenv("R2_ENDPOINT", "http://127.0.0.1:1") + t.Setenv("R2_ACCESS_KEY_ID", "k") + t.Setenv("R2_SECRET_ACCESS_KEY", "s") + t.Setenv("VALKEY_ADDR", "127.0.0.1:6379") + + c, err := Load() + require.NoError(t, err) + require.Equal(t, "https://.freecode.camp", c.Aliases.ProductionURLFormat) + require.Equal(t, "https://.preview.freecode.camp", c.Aliases.PreviewURLFormat) +} + +func TestLoad_PublicURLFormatsAreOverridable(t *testing.T) { + t.Setenv("GH_CLIENT_ID", "cid") + t.Setenv("JWT_SIGNING_KEY", "0123456789abcdef0123456789abcdef") + t.Setenv("R2_ENDPOINT", "http://127.0.0.1:1") + t.Setenv("R2_ACCESS_KEY_ID", "k") + t.Setenv("R2_SECRET_ACCESS_KEY", "s") + t.Setenv("VALKEY_ADDR", "127.0.0.1:6379") + t.Setenv("PUBLIC_URL_PRODUCTION_FORMAT", "https://.example.test") + t.Setenv("PUBLIC_URL_PREVIEW_FORMAT", "https://.pre.example.test") + + c, err := Load() + require.NoError(t, err) + require.Equal(t, "https://.example.test", c.Aliases.ProductionURLFormat) + require.Equal(t, "https://.pre.example.test", c.Aliases.PreviewURLFormat) +} + +func TestLoad_RejectsAPublicURLFormatWithoutSiteToken(t *testing.T) { + t.Setenv("GH_CLIENT_ID", "cid") + t.Setenv("JWT_SIGNING_KEY", "0123456789abcdef0123456789abcdef") + t.Setenv("R2_ENDPOINT", "http://127.0.0.1:1") + t.Setenv("R2_ACCESS_KEY_ID", "k") + t.Setenv("R2_SECRET_ACCESS_KEY", "s") + t.Setenv("VALKEY_ADDR", "127.0.0.1:6379") + t.Setenv("PUBLIC_URL_PRODUCTION_FORMAT", "https://freecode.camp") + + _, err := Load() + require.ErrorContains(t, err, "", + "a public URL that ignores the site would hand every deploy the same wrong link") +} + +func TestLoad_BlastCapDefaultsToARealCeiling(t *testing.T) { + t.Setenv("GH_CLIENT_ID", "cid") + t.Setenv("JWT_SIGNING_KEY", "0123456789abcdef0123456789abcdef") + t.Setenv("R2_ENDPOINT", "http://127.0.0.1:1") + t.Setenv("R2_ACCESS_KEY_ID", "k") + t.Setenv("R2_SECRET_ACCESS_KEY", "s") + t.Setenv("VALKEY_ADDR", "127.0.0.1:6379") + + c, err := Load() + require.NoError(t, err) + require.Equal(t, 10, c.Cleanup.BlastCap, + "an absent CLEANUP_BLAST_CAP once left the cap at 0, which both consumers read as unlimited") +} diff --git a/internal/handler/deploy.go b/internal/handler/deploy.go index ad799c5..f0bc987 100644 --- a/internal/handler/deploy.go +++ b/internal/handler/deploy.go @@ -372,13 +372,10 @@ func (h *Handlers) aliasKey(site, mode string) string { // publicURL returns the user-visible URL for a finalized deploy. func (h *Handlers) publicURL(site, mode string) string { - if h.PublicURLForSite != nil { - return h.PublicURLForSite(site, mode) - } if mode == "production" { - return "https://" + site + ".freecode.camp" + return strings.ReplaceAll(h.PublicProductionURLFmt, "", site) } - return "https://" + site + ".preview.freecode.camp" + return strings.ReplaceAll(h.PublicPreviewURLFmt, "", site) } // normalizeMode validates and normalizes finalize/promote/rollback `mode` arg. diff --git a/internal/handler/handler.go b/internal/handler/handler.go index 252c9d7..6146704 100644 --- a/internal/handler/handler.go +++ b/internal/handler/handler.go @@ -116,14 +116,17 @@ type Handlers struct { R2 R2Store AliasProductionFmt string // e.g. "/production" AliasPreviewFmt string // e.g. "/preview" - Tombstones TombstoneStore - TrashPrefixBase string // e.g. "_trash/" - Trash TrashStore - TrashRecovery time.Duration - Outbox SiteChangeEmitter - Index DeployIndexWriter - Locker SiteLocker - Audit AuditStore + + PublicProductionURLFmt string // e.g. "https://.freecode.camp" + PublicPreviewURLFmt string // e.g. "https://.preview.freecode.camp" + Tombstones TombstoneStore + TrashPrefixBase string // e.g. "_trash/" + Trash TrashStore + TrashRecovery time.Duration + Outbox SiteChangeEmitter + Index DeployIndexWriter + Locker SiteLocker + Audit AuditStore // DeployPrefix is the parsed deploy-key template. DeployPrefix DeployPrefixTemplate // UploadMaxBytes caps a single PUT /upload body size. 0 or @@ -149,7 +152,6 @@ type Handlers struct { AuditReadAuthzTeam string NewDeployID func(sha string) string Now func() time.Time - PublicURLForSite func(site, mode string) string // e.g. preview → "https://www.preview.freecode.camp" readyzValkey probeState readyzR2 probeState diff --git a/internal/handler/publicurl_test.go b/internal/handler/publicurl_test.go new file mode 100644 index 0000000..0424041 --- /dev/null +++ b/internal/handler/publicurl_test.go @@ -0,0 +1,20 @@ +package handler + +import ( + "testing" + + "github.com/stretchr/testify/require" +) + +func TestPublicURL_RendersFromTheConfiguredFormats(t *testing.T) { + t.Parallel() + + h := &Handlers{ + PublicProductionURLFmt: "https://.example.test", + PublicPreviewURLFmt: "https://.pre.example.test", + } + + require.Equal(t, "https://www.example.test", h.publicURL("www", "production")) + require.Equal(t, "https://www.pre.example.test", h.publicURL("www", "preview"), + "the URL handed to the CLI must come from config, not a constant baked into the binary") +} diff --git a/internal/handler/test_helpers_test.go b/internal/handler/test_helpers_test.go index 31d3b99..3d2f7da 100644 --- a/internal/handler/test_helpers_test.go +++ b/internal/handler/test_helpers_test.go @@ -501,13 +501,9 @@ func newTestHandlers(t *testing.T, gh *fakeGH, st *fakeSites, store R2Store) (*H NewDeployID: func(sha string) string { return "20260420-141522-" + sha[:min(7, len(sha))] }, - Now: time.Now, - PublicURLForSite: func(site, mode string) string { - if mode == "production" { - return "https://" + site + ".freecode.camp" - } - return "https://" + site + ".preview.freecode.camp" - }, + Now: time.Now, + PublicProductionURLFmt: "https://.freecode.camp", + PublicPreviewURLFmt: "https://.preview.freecode.camp", } return h, jwt } From 7e267300fb3ca9c5929d391b8a61bcd32e24b9fc Mon Sep 17 00:00:00 2001 From: Mrugesh Mohapatra Date: Mon, 17 Aug 2026 10:01:36 +0530 Subject: [PATCH 08/41] feat(pg): record a pending row when a deploy starts --- internal/pg/pending.go | 44 +++++++++++++++ internal/pg/repo_pending_test.go | 94 ++++++++++++++++++++++++++++++++ 2 files changed, 138 insertions(+) create mode 100644 internal/pg/pending.go create mode 100644 internal/pg/repo_pending_test.go diff --git a/internal/pg/pending.go b/internal/pg/pending.go new file mode 100644 index 0000000..759477f --- /dev/null +++ b/internal/pg/pending.go @@ -0,0 +1,44 @@ +package pg + +import ( + "context" + "fmt" + "time" + + "github.com/freeCodeCamp/artemis/internal/gc" +) + +const StatePending = "pending" + +func (r *Repo) BeginDeploy(ctx context.Context, site, id string, mtime time.Time) error { + _, err := r.pool.Exec(ctx, ` + INSERT INTO deploys (site, id, mtime, bytes, has_marker, state) + VALUES ($1, $2, $3, 0, false, $4) + ON CONFLICT (site, id) DO NOTHING`, + site, id, mtime, StatePending) + if err != nil { + return fmt.Errorf("pg begin deploy %s/%s: %w", site, id, err) + } + return nil +} + +func (r *Repo) ExpiredPendingDeploys(ctx context.Context, site string, before time.Time) ([]gc.Deploy, error) { + rows, err := r.pool.Query(ctx, ` + SELECT id, mtime, bytes FROM deploys + WHERE site = $1 AND state = $2 AND mtime < $3 + ORDER BY mtime`, site, StatePending, before) + if err != nil { + return nil, fmt.Errorf("pg expired pending deploys %s: %w", site, err) + } + defer rows.Close() + + var out []gc.Deploy + for rows.Next() { + var d gc.Deploy + if err := rows.Scan(&d.ID, &d.Mtime, &d.Bytes); err != nil { + return nil, fmt.Errorf("pg scan pending deploy %s: %w", site, err) + } + out = append(out, d) + } + return out, rows.Err() +} diff --git a/internal/pg/repo_pending_test.go b/internal/pg/repo_pending_test.go new file mode 100644 index 0000000..ba4c464 --- /dev/null +++ b/internal/pg/repo_pending_test.go @@ -0,0 +1,94 @@ +package pg + +import ( + "context" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func TestRepo_BeginDeploy_IsInvisibleToEveryActiveRead(t *testing.T) { + repo := newTestRepo(t) + ctx := context.Background() + t0 := time.Now().UTC().Truncate(time.Second) + + require.NoError(t, repo.BeginDeploy(ctx, "www", "d1", t0)) + + deploys, err := repo.DeploysForSite(ctx, "www") + require.NoError(t, err) + assert.Empty(t, deploys, + "a deploy that has only been initialised holds no finished bytes, so retention planning, the drift "+ + "denominator and the API must all keep ignoring it") + + n, err := repo.CountDeploys(ctx) + require.NoError(t, err) + assert.Zero(t, n) +} + +func TestRepo_BeginDeploy_IsPromotedByFinalize(t *testing.T) { + repo := newTestRepo(t) + ctx := context.Background() + t0 := time.Now().UTC().Truncate(time.Second) + + require.NoError(t, repo.BeginDeploy(ctx, "www", "d1", t0)) + require.NoError(t, repo.FinalizeAtomic(ctx, "www", "d1", "production", t0, 4096)) + + deploys, err := repo.DeploysForSite(ctx, "www") + require.NoError(t, err) + require.Len(t, deploys, 1, + "finalize already upserts ON CONFLICT ... SET state = 'active', so the pending row it lands on must "+ + "become the ordinary active deploy with no extra write") + assert.EqualValues(t, 4096, deploys[0].Bytes) + + pending, err := repo.ExpiredPendingDeploys(ctx, "www", t0.Add(time.Hour)) + require.NoError(t, err) + assert.Empty(t, pending, "a finalized deploy must never be reaped as an abandoned one") +} + +func TestRepo_BeginDeploy_IsIdempotentAndNeverDemotesALiveDeploy(t *testing.T) { + repo := newTestRepo(t) + ctx := context.Background() + t0 := time.Now().UTC().Truncate(time.Second) + + require.NoError(t, repo.BeginDeploy(ctx, "www", "d1", t0)) + require.NoError(t, repo.FinalizeAtomic(ctx, "www", "d1", "production", t0, 4096)) + require.NoError(t, repo.BeginDeploy(ctx, "www", "d1", t0)) + + deploys, err := repo.DeploysForSite(ctx, "www") + require.NoError(t, err) + require.Len(t, deploys, 1, + "a retried init against a finalized id must not flip a serving deploy back to pending, which would "+ + "queue the live site's own bytes for expiry") + assert.EqualValues(t, 4096, deploys[0].Bytes) +} + +func TestRepo_ExpiredPendingDeploys_ReturnsOnlyRowsPastTheCutoff(t *testing.T) { + repo := newTestRepo(t) + ctx := context.Background() + t0 := time.Now().UTC().Truncate(time.Second) + + require.NoError(t, repo.BeginDeploy(ctx, "www", "old", t0.Add(-96*time.Hour))) + require.NoError(t, repo.BeginDeploy(ctx, "www", "fresh", t0)) + require.NoError(t, repo.BeginDeploy(ctx, "learn", "other", t0.Add(-96*time.Hour))) + + got, err := repo.ExpiredPendingDeploys(ctx, "www", t0.Add(-72*time.Hour)) + require.NoError(t, err) + require.Len(t, got, 1, + "the cutoff must sit far above the deploy JWT TTL so an upload still in flight is never reaped") + assert.Equal(t, "old", got[0].ID) +} + +func TestRepo_ExpiredPendingDeploys_IgnoresActiveRows(t *testing.T) { + repo := newTestRepo(t) + ctx := context.Background() + t0 := time.Now().UTC().Truncate(time.Second) + + require.NoError(t, repo.UpsertDeploy(ctx, "www", "d1", t0.Add(-96*time.Hour), 100, true, "active")) + + got, err := repo.ExpiredPendingDeploys(ctx, "www", t0.Add(-72*time.Hour)) + require.NoError(t, err) + assert.Empty(t, got, + "expiry reaps abandoned sessions only; an old active deploy is retention's business, not this query's") +} From 1059c6d80b49f66a883a911cb3d11eeafec90263 Mon Sep 17 00:00:00 2001 From: Mrugesh Mohapatra Date: Mon, 17 Aug 2026 10:01:37 +0530 Subject: [PATCH 09/41] feat(handler): register the deploy session at init --- cmd/artemis/gcwire.go | 2 + internal/handler/deploy.go | 1 + internal/handler/deploy_pending_test.go | 85 +++++++++++++++++++++++++ internal/handler/handler.go | 25 ++++++++ 4 files changed, 113 insertions(+) create mode 100644 internal/handler/deploy_pending_test.go diff --git a/cmd/artemis/gcwire.go b/cmd/artemis/gcwire.go index 8e86a94..b9ee844 100644 --- a/cmd/artemis/gcwire.go +++ b/cmd/artemis/gcwire.go @@ -43,6 +43,7 @@ func wirePGRepo(h *handler.Handlers, repo *pg.Repo) { h.Tombstones = repo h.Trash = repo h.Index = repo + h.Pending = repo h.Locker = repo h.Audit = repo } @@ -212,6 +213,7 @@ func newGCWiring(cfg *config.Config, repo *pg.Repo, r2c *r2.Client) (*gcWiring, Mover: r2c, Locker: repo, LiveAliases: liveAliases, + Pending: repo, Policy: gcPolicy(cfg.Cleanup), BlastCap: cfg.Cleanup.BlastCap, DeployPrefix: layout.deployPrefix, diff --git a/internal/handler/deploy.go b/internal/handler/deploy.go index f0bc987..5fe54d8 100644 --- a/internal/handler/deploy.go +++ b/internal/handler/deploy.go @@ -86,6 +86,7 @@ func (h *Handlers) DeployInit(w http.ResponseWriter, r *http.Request) { } telemetry.FromContext(r.Context()).SetResource(req.Site, deployID) + h.beginPendingDeploy(r.Context(), h.DeployPrefix.SiteDirname(req.Site), deployID) h.logAction(r.Context(), "deploy.init", "success") h.auditFromScope(r.Context(), "deploy.init", "success", map[string]any{"sha": req.SHA}) diff --git a/internal/handler/deploy_pending_test.go b/internal/handler/deploy_pending_test.go new file mode 100644 index 0000000..3c3db67 --- /dev/null +++ b/internal/handler/deploy_pending_test.go @@ -0,0 +1,85 @@ +package handler + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "net/http" + "net/http/httptest" + "sync" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +type recordingBeginner struct { + mu sync.Mutex + calls [][2]string + err error +} + +func (b *recordingBeginner) BeginDeploy(_ context.Context, site, id string, _ time.Time) error { + b.mu.Lock() + defer b.mu.Unlock() + b.calls = append(b.calls, [2]string{site, id}) + return b.err +} + +func newPendingHandlers(t *testing.T) *Handlers { + t.Helper() + gh := &fakeGH{ + tokenLogins: map[string]string{"tok": "alice"}, + userTeams: map[string]map[string]bool{"alice": {"team-eng": true}}, + } + h, _ := newTestHandlers(t, gh, standardSites(), newFakeR2()) + return h +} + +func initRequest(t *testing.T, h *Handlers) *httptest.ResponseRecorder { + t.Helper() + body, _ := json.Marshal(DeployInitRequest{Site: "www", SHA: "abc1234567"}) + r := httptest.NewRequest(http.MethodPost, "/api/deploy/init", bytes.NewReader(body)). + WithContext(contextWithLogin(context.Background(), "alice", "tok")) + w := httptest.NewRecorder() + h.DeployInit(w, r) + return w +} + +func TestDeployInit_RecordsThePendingDeployInTheStorageKeyspace(t *testing.T) { + h := newPendingHandlers(t) + beginner := &recordingBeginner{} + h.Pending = beginner + + rec := initRequest(t, h) + require.Equal(t, http.StatusOK, rec.Code) + + require.Len(t, beginner.calls, 1, + "an init that records nothing leaves any bytes the client then uploads unowned by every reaper, "+ + "which is the whole orphan class reconcile exists to scan for") + assert.Equal(t, h.DeployPrefix.SiteDirname("www"), beginner.calls[0][0], + "finalize upserts ON CONFLICT (site, id) using SiteDirname, so a pending row written under the slug "+ + "would never be promoted and would be reaped while the deploy is live") +} + +func TestDeployInit_SucceedsWhenThePendingWriteFails(t *testing.T) { + h := newPendingHandlers(t) + h.Pending = &recordingBeginner{err: errors.New("pg down")} + + rec := initRequest(t, h) + + assert.Equal(t, http.StatusOK, rec.Code, + "the index is optional wiring (deploy-only mode runs without one), so a bookkeeping write must never "+ + "turn a database blip into a total deploy outage") +} + +func TestDeployInit_SucceedsWithNoPendingWriterWired(t *testing.T) { + h := newPendingHandlers(t) + h.Pending = nil + + rec := initRequest(t, h) + + assert.Equal(t, http.StatusOK, rec.Code) +} diff --git a/internal/handler/handler.go b/internal/handler/handler.go index 6146704..8417cab 100644 --- a/internal/handler/handler.go +++ b/internal/handler/handler.go @@ -90,6 +90,10 @@ type DeployIndexWriter interface { AliasAtomic(ctx context.Context, site, name, deployID string, at time.Time) error } +type PendingDeployWriter interface { + BeginDeploy(ctx context.Context, site, deployID string, mtime time.Time) error +} + type SiteLocker interface { WithSiteLock(ctx context.Context, site string, fn func() error) error } @@ -125,6 +129,7 @@ type Handlers struct { TrashRecovery time.Duration Outbox SiteChangeEmitter Index DeployIndexWriter + Pending PendingDeployWriter Locker SiteLocker Audit AuditStore // DeployPrefix is the parsed deploy-key template. @@ -216,6 +221,26 @@ func (h *Handlers) audit(ctx context.Context, e pg.AuditEvent) { } } +const pendingWriteTimeout = 5 * time.Second + +func (h *Handlers) beginPendingDeploy(ctx context.Context, site, deployID string) { + if h.Pending == nil { + return + } + beginCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), pendingWriteTimeout) + defer cancel() + if err := h.Pending.BeginDeploy(beginCtx, site, deployID, h.Now().UTC()); err != nil { + slog.ErrorContext(ctx, "deploy.pending.write_failed", "site", site, "deploy_id", deployID, "err", err) + if hub := sentry.GetHubFromContext(ctx); hub != nil { + hub.WithScope(func(scope *sentry.Scope) { + scope.SetTag("op", "deploy.pending") + scope.SetFingerprint([]string{"deploy.pending"}) + hub.CaptureException(err) + }) + } + } +} + func (h *Handlers) logAction(ctx context.Context, action, outcome string, attrs ...slog.Attr) { sc := telemetry.FromContext(ctx) sc.SetAction(action) From 7a92fde55206ef81a5ca1673a4bef93f6fc21e47 Mon Sep 17 00:00:00 2001 From: Mrugesh Mohapatra Date: Mon, 17 Aug 2026 10:01:37 +0530 Subject: [PATCH 10/41] feat(gc): expire abandoned deploy sessions --- internal/gc/gcsite.go | 18 +++++++ internal/gc/gcsite_pending_test.go | 86 ++++++++++++++++++++++++++++++ internal/gc/gcsite_test.go | 2 +- internal/gc/plan.go | 7 +++ internal/gc/plan_test.go | 25 +++++++-- internal/gc/retain.go | 1 + 6 files changed, 134 insertions(+), 5 deletions(-) create mode 100644 internal/gc/gcsite_pending_test.go diff --git a/internal/gc/gcsite.go b/internal/gc/gcsite.go index bc8c2ea..512b670 100644 --- a/internal/gc/gcsite.go +++ b/internal/gc/gcsite.go @@ -15,6 +15,10 @@ type Store interface { Tombstone(ctx context.Context, site string, d Deploy) error } +type PendingSource interface { + ExpiredPendingDeploys(ctx context.Context, site string, before time.Time) ([]Deploy, error) +} + type Mover interface { MovePrefix(ctx context.Context, src, dst string) (int, error) } @@ -43,6 +47,7 @@ type SiteGC struct { LiveAliases func(ctx context.Context, site string) (map[string]struct{}, error) Now func() time.Time Audit GCAuditor + Pending PendingSource } type GCResult struct { @@ -56,6 +61,13 @@ type GCResult struct { DryRun bool } +func (g *SiteGC) expiredPending(ctx context.Context, site string) ([]Deploy, error) { + if g.Pending == nil { + return nil, nil + } + return g.Pending.ExpiredPendingDeploys(ctx, site, g.Now().Add(-g.Policy.Grace)) +} + func (g *SiteGC) Run(ctx context.Context, site string, dryRun bool) (GCResult, error) { res := GCResult{Site: site, DryRun: dryRun} @@ -68,8 +80,14 @@ func (g *SiteGC) Run(ctx context.Context, site string, dryRun bool) (GCResult, e return res, fmt.Errorf("gc %s: load aliases: %w", site, err) } + expired, err := g.expiredPending(ctx, site) + if err != nil { + return res, fmt.Errorf("gc %s: load pending deploys: %w", site, err) + } + plan := PlanSite(site, RetainInput{ Deploys: deploys, + Expired: expired, AliasTargets: targets, LastAliasChange: lastChange, Now: g.Now(), diff --git a/internal/gc/gcsite_pending_test.go b/internal/gc/gcsite_pending_test.go new file mode 100644 index 0000000..16b2f87 --- /dev/null +++ b/internal/gc/gcsite_pending_test.go @@ -0,0 +1,86 @@ +package gc + +import ( + "context" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +type fakePending struct { + rows map[string][]Deploy + cuts []time.Time + err error +} + +func (p *fakePending) ExpiredPendingDeploys(_ context.Context, site string, before time.Time) ([]Deploy, error) { + p.cuts = append(p.cuts, before) + if p.err != nil { + return nil, p.err + } + return p.rows[site], nil +} + +func pendingSiteGC(t *testing.T, store Store, mover Mover, pending PendingSource) *SiteGC { + t.Helper() + g := newSiteGC(store, mover) + g.Pending = pending + return g +} + +func TestSiteGC_ExpiresAnAbandonedDeploySession(t *testing.T) { + store := &fakeStore{deploys: map[string][]Deploy{}} + mover := &fakeMover{} + pending := &fakePending{rows: map[string][]Deploy{ + "www": {{ID: "abandoned", Mtime: testNow.Add(-96 * time.Hour)}}, + }} + + res, err := pendingSiteGC(t, store, mover, pending).Run(context.Background(), "www", false) + require.NoError(t, err) + + assert.Equal(t, []string{"www/abandoned"}, store.tombstoned, + "an init that uploaded bytes and never finalized is the orphan class reconcile had to scan R2 to "+ + "find; expiring the pending row collects it through the index instead") + require.Len(t, mover.moves, 1) + assert.Equal(t, [2]string{"www/deploys/abandoned/", "_trash/www/abandoned/"}, mover.moves[0]) + assert.Contains(t, res.Tombstoned, "abandoned") +} + +func TestSiteGC_ExpiryCutoffSitsAtTheGraceWindow(t *testing.T) { + pending := &fakePending{rows: map[string][]Deploy{}} + + _, err := pendingSiteGC(t, &fakeStore{deploys: map[string][]Deploy{}}, &fakeMover{}, pending). + Run(context.Background(), "www", false) + require.NoError(t, err) + + require.Len(t, pending.cuts, 1) + assert.Equal(t, testNow.Add(-testPolicy().Grace), pending.cuts[0], + "the cutoff must stay far above the 15-minute deploy JWT TTL, or a session still uploading is reaped "+ + "out from under a live client") +} + +func TestSiteGC_DryRunReportsPendingWithoutTouchingAnything(t *testing.T) { + store := &fakeStore{deploys: map[string][]Deploy{}} + mover := &fakeMover{} + pending := &fakePending{rows: map[string][]Deploy{ + "www": {{ID: "abandoned", Mtime: testNow.Add(-96 * time.Hour)}}, + }} + + res, err := pendingSiteGC(t, store, mover, pending).Run(context.Background(), "www", true) + require.NoError(t, err) + + assert.Contains(t, res.Planned, "abandoned") + assert.Empty(t, store.tombstoned, "a dry run must not write") + assert.Empty(t, mover.moves, "a dry run must not move bytes") +} + +func TestSiteGC_RunsWithoutAPendingSourceWired(t *testing.T) { + store := &fakeStore{deploys: map[string][]Deploy{"www": sixOld()}} + + res, err := newSiteGC(store, &fakeMover{}).Run(context.Background(), "www", false) + + require.NoError(t, err, "pending expiry is optional wiring; its absence must not break retention") + assert.NotEmpty(t, res.Tombstoned) +} diff --git a/internal/gc/gcsite_test.go b/internal/gc/gcsite_test.go index 8dc095f..0def222 100644 --- a/internal/gc/gcsite_test.go +++ b/internal/gc/gcsite_test.go @@ -78,7 +78,7 @@ func newSiteGC(store Store, mover Mover) *SiteGC { Store: store, Mover: mover, Policy: testPolicy(), - BlastCap: 0, + BlastCap: defaultTestBlastCap, Locker: &fakeLocker{}, DeployPrefix: func(site, id string) string { return site + "/deploys/" + id + "/" }, TrashPrefix: func(site, id string) string { return "_trash/" + site + "/" + id + "/" }, diff --git a/internal/gc/plan.go b/internal/gc/plan.go index 42824ec..8de1b8c 100644 --- a/internal/gc/plan.go +++ b/internal/gc/plan.go @@ -12,8 +12,15 @@ type Plan struct { func PlanSite(site string, in RetainInput, p Policy, blastCap int) Plan { _, del := Retain(in, p) + del = append(del, in.Expired...) plan := Plan{Site: site} + if len(del) > 0 && blastCap <= 0 { + plan.Aborted = true + plan.Reason = fmt.Sprintf( + "refusing %d deletes: blast-cap 0 means no ceiling was configured", len(del)) + return plan + } if blastCap > 0 && len(del) > blastCap { plan.Aborted = true plan.Reason = fmt.Sprintf("delete plan of %d exceeds blast-cap %d; reaping oldest %d this run", len(del), blastCap, blastCap) diff --git a/internal/gc/plan_test.go b/internal/gc/plan_test.go index 84a3183..390e4c2 100644 --- a/internal/gc/plan_test.go +++ b/internal/gc/plan_test.go @@ -21,7 +21,7 @@ func oldDeploys(n int, eachBytes int64) []Deploy { } func TestPlanSite_KeepN(t *testing.T) { - plan := PlanSite("www", RetainInput{Deploys: oldDeploys(6, 100), Now: testNow}, testPolicy(), 0) + plan := PlanSite("www", RetainInput{Deploys: oldDeploys(6, 100), Now: testNow}, testPolicy(), defaultTestBlastCap) assert.Len(t, plan.Delete, 3, "6 old deploys, keepN=3 -> 3 deletable (V2)") assert.False(t, plan.Aborted) @@ -47,8 +47,25 @@ func TestGC_BlastCap(t *testing.T) { assert.False(t, ids["d-old"], "newest deletable deploy is spared until a later run") } -func TestPlanSite_BlastCapDisabled(t *testing.T) { +func TestPlanSite_BlastCapZeroRefusesEveryDelete(t *testing.T) { plan := PlanSite("www", RetainInput{Deploys: oldDeploys(20, 1), Now: testNow}, testPolicy(), 0) - assert.False(t, plan.Aborted, "blastCap=0 disables the cap") - assert.Len(t, plan.Delete, 17) + assert.True(t, plan.Aborted, + "an unset cap once meant unlimited, so an environment that forgot CLEANUP_BLAST_CAP reaped a whole "+ + "site in one run") + assert.Empty(t, plan.Delete) + assert.Contains(t, plan.Reason, "blast-cap 0") +} + +func TestPlanSite_ExpiredPendingJoinsTheDeleteSetUnderTheSameCap(t *testing.T) { + in := RetainInput{ + Deploys: oldDeploys(6, 100), + Expired: []Deploy{{ID: "abandoned", Mtime: testNow.Add(-96 * time.Hour)}}, + Now: testNow, + } + + plan := PlanSite("www", in, testPolicy(), 2) + + assert.True(t, plan.Aborted, "3 retention deletes plus 1 expired pending exceeds a cap of 2") + assert.Len(t, plan.Delete, 2, + "abandoned sessions must be bounded by the same ceiling as retention, not appended past it") } diff --git a/internal/gc/retain.go b/internal/gc/retain.go index ad59ac0..c3f029e 100644 --- a/internal/gc/retain.go +++ b/internal/gc/retain.go @@ -22,6 +22,7 @@ type Policy struct { type RetainInput struct { Deploys []Deploy + Expired []Deploy AliasTargets map[string]struct{} LastAliasChange time.Time Now time.Time From a5884ce4a5190a75c4068767864a0434270ab9aa Mon Sep 17 00:00:00 2001 From: Mrugesh Mohapatra Date: Mon, 17 Aug 2026 10:01:37 +0530 Subject: [PATCH 11/41] fix(drift): alert on accruing reclaimable drift --- cmd/artemis/driftalert.go | 12 +++++ cmd/artemis/driftalert_threshold_test.go | 56 +++++++++++++++++++++++ internal/observability/cronshaped_test.go | 25 ++++++++++ internal/observability/sentry.go | 8 +++- 4 files changed, 99 insertions(+), 2 deletions(-) create mode 100644 cmd/artemis/driftalert_threshold_test.go create mode 100644 internal/observability/cronshaped_test.go diff --git a/cmd/artemis/driftalert.go b/cmd/artemis/driftalert.go index e3893fd..e41f2b2 100644 --- a/cmd/artemis/driftalert.go +++ b/cmd/artemis/driftalert.go @@ -13,8 +13,11 @@ const ( opDriftSelfCheck = "drift.selfcheck" opDriftUnreadable = "drift.unreadable" opDriftAliasedMissing = "drift.aliased_missing" + opDriftReclaimable = "drift.reclaimable" ) +const reclaimableAlertThreshold = 25 + type driftVerdict struct { Op string Err error @@ -38,6 +41,15 @@ func classifyDrift(res sweepResult) driftVerdict { if unread != nil { return driftVerdict{Op: opDriftUnreadable, Err: unread, Fails: true} } + if reindex, tombstone, _, _ := res.totals(); reindex+tombstone >= reclaimableAlertThreshold { + return driftVerdict{ + Op: opDriftReclaimable, + Err: fmt.Errorf( + "%d deploys are reclaimable across %d sites (>= %d): storage is accruing faster than it is "+ + "collected; run `artemis reconcile --apply` and find what stopped expiring", + reindex+tombstone, res.Stats.Sites, reclaimableAlertThreshold), + } + } return driftVerdict{} } diff --git a/cmd/artemis/driftalert_threshold_test.go b/cmd/artemis/driftalert_threshold_test.go new file mode 100644 index 0000000..4642a1f --- /dev/null +++ b/cmd/artemis/driftalert_threshold_test.go @@ -0,0 +1,56 @@ +package main + +import ( + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func sweepWithReclaimable(reindex, tombstone int) sweepResult { + var s siteDrift + s.Site = "www.freecode.camp" + for i := 0; i < reindex; i++ { + s.Reindex = append(s.Reindex, "r") + } + for i := 0; i < tombstone; i++ { + s.Tombstone = append(s.Tombstone, "t") + } + return sweepResult{ + Reports: []siteDrift{s}, + Stats: sweepStats{Sites: 1, PGDeploys: 1, IndexedTotal: 1, R2Objects: 1}, + } +} + +func TestClassifyDrift_StaysQuietBelowTheReclaimableThreshold(t *testing.T) { + t.Parallel() + + v := classifyDrift(sweepWithReclaimable(reclaimableAlertThreshold-1, 0)) + + assert.Empty(t, v.Op, "a handful of reclaimable items is the expected residue of an interrupted deploy") +} + +func TestClassifyDrift_AlertsOnceReclaimableDriftAccumulates(t *testing.T) { + t.Parallel() + + v := classifyDrift(sweepWithReclaimable(reclaimableAlertThreshold, 0)) + + require.Equal(t, opDriftReclaimable, v.Op, + "abandoned deploy sessions accrued for 33 days with nothing watching; a report-only cron that never "+ + "raises on them is indistinguishable from not looking") + assert.False(t, v.Fails, "reclaimable drift is storage cost, not an outage: alert, do not fail the run") + require.Error(t, v.Err) + assert.Contains(t, v.Err.Error(), "artemis reconcile") +} + +func TestClassifyDrift_AliasedMissingOutranksTheReclaimableThreshold(t *testing.T) { + t.Parallel() + + res := sweepWithReclaimable(reclaimableAlertThreshold+50, 0) + res.Reports[0].Aliased = []string{"d1"} + + v := classifyDrift(res) + + assert.Equal(t, opDriftAliasedMissing, v.Op, + "a live site serving nothing must not be masked by a large but harmless reclaimable count") +} diff --git a/internal/observability/cronshaped_test.go b/internal/observability/cronshaped_test.go new file mode 100644 index 0000000..ed55796 --- /dev/null +++ b/internal/observability/cronshaped_test.go @@ -0,0 +1,25 @@ +package observability + +import ( + "testing" + + "github.com/stretchr/testify/assert" +) + +func TestCronShapedOps_CoverEveryNightlyDriftVerdict(t *testing.T) { + t.Parallel() + + for _, op := range []string{ + "drift.sweep", + "drift.selfcheck", + "drift.unreadable", + "drift.aliased_missing", + "drift.reclaimable", + "tombstone.purge", + } { + assert.True(t, cronShapedOps[op], + "%s fires at most once a night, so the transient-rate tracker (threshold 3, 26h reset) would "+ + "swallow it for two nights before escalating — and an in-memory count resets on every pod "+ + "restart, so it may never escalate at all", op) + } +} diff --git a/internal/observability/sentry.go b/internal/observability/sentry.go index 1a1f336..228804d 100644 --- a/internal/observability/sentry.go +++ b/internal/observability/sentry.go @@ -375,8 +375,12 @@ func NewSlogHandler(minLevel slog.Level) slog.Handler { } var cronShapedOps = map[string]bool{ - "drift.sweep": true, - "tombstone.purge": true, + "drift.sweep": true, + "drift.selfcheck": true, + "drift.unreadable": true, + "drift.aliased_missing": true, + "drift.reclaimable": true, + "tombstone.purge": true, } // CaptureBackground reports an error raised outside any HTTP request From 183bdb659475d1b7800c4a02c79c8a93bb9fd0c9 Mon Sep 17 00:00:00 2001 From: Mrugesh Mohapatra Date: Mon, 17 Aug 2026 10:02:14 +0530 Subject: [PATCH 12/41] test(gc): pin both key renderers to one layout --- cmd/artemis/reconcile_keyspace_test.go | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/cmd/artemis/reconcile_keyspace_test.go b/cmd/artemis/reconcile_keyspace_test.go index 244579e..aa73013 100644 --- a/cmd/artemis/reconcile_keyspace_test.go +++ b/cmd/artemis/reconcile_keyspace_test.go @@ -59,3 +59,26 @@ func TestStorageSiteNames_BareFormatIsIdentity(t *testing.T) { require.Equal(t, []string{"test", "www"}, names) require.Equal(t, tmpl.SitePrefix("test"), layout.sitePrefix(names[0])) } + +func TestGCLayout_AgreesWithTheWritePathOnEveryRenderedKey(t *testing.T) { + t.Parallel() + + tmpl, err := handler.NewDeployPrefixTemplate(domainFormat) + require.NoError(t, err) + layout, err := newGCLayout(domainFormat, "_trash/") + require.NoError(t, err) + + for _, slug := range []string{"test", "hello-universe", "www"} { + dirname := tmpl.SiteDirname(slug) + const id = "20260101-000000-abc1234" + + require.Equal(t, tmpl.SitePrefix(slug), layout.sitePrefix(dirname), + "slug %q: the sweep lists a prefix the write path never produces", slug) + require.Equal(t, tmpl.DeployPrefix(slug, id), layout.deployPrefix(dirname, id), + "slug %q: gc would move a prefix no deploy was written to, so the real bytes stay and the "+ + "tombstone dates nothing", slug) + require.Equal(t, "_trash/"+dirname+"/"+id+"/", layout.trashPrefix(dirname, id), + "slug %q: tombstone-purge hard-deletes _trash/// by reconstructing it from the "+ + "tombstone row, so any other shape leaks bytes forever", slug) + } +} From a779ece51fa42b3662b7a5670d5f85d4f27c1613 Mon Sep 17 00:00:00 2001 From: Mrugesh Mohapatra Date: Mon, 17 Aug 2026 10:03:07 +0530 Subject: [PATCH 13/41] docs: record what the drift sprint shipped --- docs/design/0005-drift-at-source.md | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/docs/design/0005-drift-at-source.md b/docs/design/0005-drift-at-source.md index 1839275..6febc35 100644 --- a/docs/design/0005-drift-at-source.md +++ b/docs/design/0005-drift-at-source.md @@ -1,6 +1,6 @@ # 0005 — Drift at source: fixing the cause, not the cleanup -Status: proposed Supersedes the framing of [0004](0004-drift-detection-and-alerting.md) §rationale. +Status: P0/P2/P3 implemented on `fix/artemis-drift-at-source`; P1 deferred (see Sequencing) Supersedes the framing of [0004](0004-drift-detection-and-alerting.md) §rationale. ## Why 0004 needs superseding @@ -196,12 +196,28 @@ Everything verified during this audit, with a decision against each. "Accept, do | 11 | `outbox` has no retention — unbounded growth | **Backlog.** Small table, slow growth, no correctness impact. Needs a purge job eventually; not part of this wave. | | 12 | Dead worker code paths | **Backlog**, cosmetic. | | 13 | `RequireScope` / latched rate limiter behaviours | **Accept, documented.** Both behave as designed; the surprise is documentation, not code. Already captured in ONBOARDING §10. | +| 15 | Every drift verdict op (`drift.selfcheck`, `drift.unreadable`, `drift.aliased_missing`) is absent from `cronShapedOps` (`internal/observability/sentry.go:377`), so `alertOnDrift`'s `captureBackground(v.Op, ...)` falls through to the transient-rate tracker: threshold 3 with a 26h reset window means a nightly alert is swallowed for two nights, and the in-memory counter resets on every pod restart, so with 3 replicas it may never escalate. **A live site serving nothing could page nobody.** | **Fixed** — all four verdict ops added, pinned by a test that fails if a new one is missed. | | 14 | `PublicURLForSite` (`internal/handler/handler.go:152`) is never assigned outside tests, so the hardcoded fallback at `deploy.go:377-382` always runs — the public URL returned to the CLI bakes `freecode.camp` and `.preview.` into the binary, while every other domain fact comes from config. There is no `ROOT_DOMAIN` setting. | **Fix with P0** (one-liner): derive the URL from the configured alias formats, or add the root domain to config. Cosmetic today, silently wrong the day the root domain or preview label changes. | ## Sequencing -P0 is one commit-sized wave and should ship on its own — four defects, all small, all independently testable, with P0-1 landed RED first so the P0-2 fix has a failing test to turn green. +Shipped together on `fix/artemis-drift-at-source` at the operator's direction — one sprint covering every bug, ten commits, each RED-first: -P1 and P2 are each their own wave. P3 is a follow-up to P2, except for the interim `reclaimable > 50` branch, which rides with P0. +| commit | phase | +| --- | --- | +| `fix(gc): read live aliases in the sweep keyspace` | P0-1 + P0-2 | +| `fix(cli): reject unknown subcommands and stray args` | P0, finding 9 | +| `fix(gc): record the tombstone row before moving bytes` | P0-4 | +| `fix(gc): make a zero blast cap refuse, not unleash` | P0-3 | +| `fix(handler): build public URLs from config` | finding 14 | +| `feat(pg): record a pending row when a deploy starts` | P2 | +| `feat(handler): register the deploy session at init` | P2 | +| `feat(gc): expire abandoned deploy sessions` | P2 | +| `fix(drift): alert on accruing reclaimable drift` | P3 + finding 15 | +| `test(gc): pin both key renderers to one layout` | P1, partial | + +**P1 is deferred, deliberately.** The `Slug`/`Dirname` type split is prevention, not a bug, and a codebase-wide mechanical refactor landing alongside a behaviour change on the deploy hot path would make the review surface unreadable. What ships instead is the cheap structural half: boot validation that the alias formats and the deploy prefix share a site segment (so the dirname reconstruction is correct by construction, not by convention), plus a test that cross-checks all three rendered keys — site, deploy, trash — between the two renderers under the production FQDN format. The type split remains the right next wave. + +The `reclaimable` threshold shipped at **25**, not the 50 originally sketched: with P2 collecting abandoned sessions at source the steady-state baseline is zero, so a lower bar is signal rather than noise. This document is the seed artifact for a new dossier. It does not belong in `artemis-audit-fixes`, which is at 19/20 with a pending converge. From 0acc548c61a1d7aa36bf5e5cb9b1506f192d1c6d Mon Sep 17 00:00:00 2001 From: Mrugesh Mohapatra Date: Mon, 17 Aug 2026 10:05:24 +0530 Subject: [PATCH 14/41] test(handler): run the pending fixture on the prod format --- internal/handler/deploy_pending_test.go | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/internal/handler/deploy_pending_test.go b/internal/handler/deploy_pending_test.go index 3c3db67..a31b33c 100644 --- a/internal/handler/deploy_pending_test.go +++ b/internal/handler/deploy_pending_test.go @@ -35,6 +35,7 @@ func newPendingHandlers(t *testing.T) *Handlers { userTeams: map[string]map[string]bool{"alice": {"team-eng": true}}, } h, _ := newTestHandlers(t, gh, standardSites(), newFakeR2()) + h.DeployPrefix = mustDeployPrefixTemplate(".freecode.camp/deploys/-/") return h } @@ -59,7 +60,10 @@ func TestDeployInit_RecordsThePendingDeployInTheStorageKeyspace(t *testing.T) { require.Len(t, beginner.calls, 1, "an init that records nothing leaves any bytes the client then uploads unowned by every reaper, "+ "which is the whole orphan class reconcile exists to scan for") - assert.Equal(t, h.DeployPrefix.SiteDirname("www"), beginner.calls[0][0], + require.Equal(t, "www.freecode.camp", h.DeployPrefix.SiteDirname("www"), + "under the default format slug and dirname are the same string, which makes the assertion below "+ + "vacuous; this fixture must run the production FQDN shape") + assert.Equal(t, "www.freecode.camp", beginner.calls[0][0], "finalize upserts ON CONFLICT (site, id) using SiteDirname, so a pending row written under the slug "+ "would never be promoted and would be reaped while the deploy is live") } From 937e315d94f85be219fbc5331c5322ebefda3f1e Mon Sep 17 00:00:00 2001 From: Mrugesh Mohapatra Date: Mon, 17 Aug 2026 10:06:31 +0530 Subject: [PATCH 15/41] docs: refresh onboarding for the drift sprint --- docs/ONBOARDING.md | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/docs/ONBOARDING.md b/docs/ONBOARDING.md index aa2f4b6..2e4a88c 100644 --- a/docs/ONBOARDING.md +++ b/docs/ONBOARDING.md @@ -152,14 +152,13 @@ The first two are *reclaimable*: unreferenced bytes and forgotten rows. The last Safety rails on the repair path: the site lock, a **second read inside the lock** before acting, a grace window so young deploys are never touched, and a blast cap. -### 7.4 The ordering inconsistency you will trip over +### 7.4 Why both reaping paths write the row first -The two reaping paths write their two side effects in **opposite orders**: +Every reaping path records the tombstone row **before** it moves the bytes — `gc-site` at `internal/gc/gcsite.go:135` then `:138`, `reconcile` at `internal/gc/reconcile.go:286` then `:290`. -- `gc-site` moves the bytes, then records the tombstone row (`internal/gc/gcsite.go:117` then `:120`) -- `reconcile` records the row, then moves the bytes (`internal/gc/reconcile.go:275` then `:279`) +That order is forced by the purge being row-driven. Bytes moved into `_trash/` without a tombstone are invisible to `tombstone-purge` (which walks `tombstones`), invisible to the index, and invisible to `reconcile` (which lists the *site* prefix, not `_trash/`) — a permanent, undetectable leak. The inverse failure is benign: a row with its bytes still at the deploy prefix shows up as ordinary reindex drift, which the nightly sweep reports and `reconcile` repairs. Both paths log `tombstone_move_deferred` when they land in that state. -Reconcile's order is the correct one: the purge is row-driven, so bytes moved without a row are a permanent leak. `gc-site` still has the leaky order — it is a known follow-up, not a fixed thing. +`gc-site` carried the leaky order until the drift-at-source sprint; if you find a doc or comment claiming otherwise, it predates that change. ______________________________________________________________________ @@ -196,15 +195,15 @@ ______________________________________________________________________ Verified traps, each a real line of code. None of these are hypothetical. -1. **`CLEANUP_BLAST_CAP` has no default.** It is absent from the defaults block (`internal/config/config.go:242-249`), so it is `0`, and both consumers treat `<= 0` as *disabled*. Production sets it to `10` explicitly; a fresh environment runs uncapped. +1. **`CLEANUP_BLAST_CAP` of `0` refuses every destructive repair.** It used to mean *unlimited*, and the code default was `0` — a safety valve that defaulted to off. It now defaults to `10` and a literal `0` is a refusal, reported as `Aborted` with a reason. Both consumers agree: `PlanSite` (`internal/gc/plan.go`) and `Reconciler.applyBlastCap` (`internal/gc/reconcile.go`). 1. **A deploy's mtime is parsed out of its ID string**, not read from R2 metadata (`internal/gc/reconcile.go:494`). An ID whose first 15 characters are not `20060102-150405` gets a zero time. 1. **The marker extends a deploy's life, it does not shorten it.** A marked deploy is kept for the full retention window; an unmarked one only for the grace window. 1. **Reconcile records `bytes = 0`** on the tombstones it creates (`internal/gc/reconcile.go:275`), so purge's "bytes reclaimed" figure under-reports. -1. **Unknown `argv` silently boots the server.** `main.go:49` and `:56` compare against two exact strings with no default case and no usage text. `artemis --help` starts a web server. -1. **`driftreport` ignores its arguments** — `main.go:50` forwards nothing, unlike `main.go:57`. `artemis driftreport --site www` sweeps the whole fleet, silently. +1. **Subcommand dispatch is closed.** `dispatchSubcommand` (`cmd/artemis/main.go:50`) returns `handled=false` only for an empty argv; anything unrecognised is an error, and `driftreport` rejects arguments outright rather than sweeping the fleet while appearing scoped. Both used to fall through — `artemis --help` once started a web server. 1. **`BACKFILL_ON_BOOT` is a different program.** `runWith` does the backfill and returns before any listener starts, so the process exits 0 having served nothing. 1. **Bare `DELETE /api/site/{slug}` removes only the registry row.** Bytes, index rows and live alias objects all survive; the site just becomes unmanaged. `?purge=true` is the destructive one. -1. **There is no "already finalized" guard on upload.** A valid JWT can keep writing into a prefix that is already the live production target, for the rest of its TTL. +1. **There is no "already finalized" guard on upload.** A valid JWT can keep writing into a prefix that is already the live production target, for the rest of its TTL. Known and accepted; see design 0005. +1. **A deploy row exists from `init`, not from `finalize`.** `deploy.init` writes `state = 'pending'` (`internal/pg/pending.go`); `FinalizeAtomic`'s existing `ON CONFLICT ... SET state = 'active'` promotes it with no extra write. Every read filters `state = 'active'`, so a pending row is invisible to retention planning, the drift denominator and the API — its only reader is `ExpiredPendingDeploys`, which `gc-site` uses to reap sessions abandoned past the grace window. The write is best-effort: a failure logs and raises to Sentry but never fails the deploy. 1. **`site-purge` writes a sentinel tombstone with `id = ''`**, and that row now blocks reindexing of *every* deploy in that site until the recovery window clears it. That is deliberate, added this week, and easy to mistake for a bug. 1. **Dead code that looks live.** `worker.RegisterDeployWorkflows` is never called outside tests — finalize, promote and rollback all run inline in the HTTP handlers. From f733478f3d66d61aa72e43f3f2358e9dfb627846 Mon Sep 17 00:00:00 2001 From: Mrugesh Mohapatra Date: Mon, 17 Aug 2026 10:13:21 +0530 Subject: [PATCH 16/41] test(cli): align boot fixtures with the alias keyspace --- cmd/artemis/bootrun_test.go | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/cmd/artemis/bootrun_test.go b/cmd/artemis/bootrun_test.go index 99ef3c9..76f7d99 100644 --- a/cmd/artemis/bootrun_test.go +++ b/cmd/artemis/bootrun_test.go @@ -52,8 +52,8 @@ func bootCfg(t *testing.T, dsn, valkeyAddr string, port int) *config.Config { cfg.GitHub.MembershipCacheTTL = time.Minute cfg.JWT.SigningKey = "0123456789abcdef0123456789abcdef" cfg.JWT.TTL = 15 * time.Minute - cfg.Aliases.ProductionKeyFormat = "/production" - cfg.Aliases.PreviewKeyFormat = "/preview" + cfg.Aliases.ProductionKeyFormat = ".example.test/production" + cfg.Aliases.PreviewKeyFormat = ".example.test/preview" cfg.Cleanup.TrashPrefix = "_trash/" cfg.Cleanup.RecoveryDays = 7 cfg.Cleanup.Grace = 72 * time.Hour @@ -332,6 +332,8 @@ func TestRun_BootsFromEnvAndExitsOnSigterm(t *testing.T) { t.Setenv("GH_CLIENT_ID", "cid") t.Setenv("JWT_SIGNING_KEY", "0123456789abcdef0123456789abcdef") t.Setenv("DEPLOY_PREFIX_FORMAT", ".example.test/deploys/-/") + t.Setenv("ALIAS_PRODUCTION_KEY_FORMAT", ".example.test/production") + t.Setenv("ALIAS_PREVIEW_KEY_FORMAT", ".example.test/preview") t.Setenv("LOG_LEVEL", "error") t.Setenv("SENTRY_DSN", "https://publickey@o0.ingest.sentry.io/0") t.Setenv("ENVIRONMENT", "test") From 96d10228a435a6f151347030e8cc4d5cec56a6c2 Mon Sep 17 00:00:00 2001 From: Mrugesh Mohapatra Date: Mon, 17 Aug 2026 10:13:22 +0530 Subject: [PATCH 17/41] fix(drift): stop the blast cap silencing the report --- internal/gc/blastcap_test.go | 14 ++++++++++++++ internal/gc/dryrun_cap_test.go | 29 +++++++++++++++++++++++++++++ internal/gc/reconcile.go | 8 +++++++- internal/gc/reconcile_race_test.go | 5 ++++- 4 files changed, 54 insertions(+), 2 deletions(-) create mode 100644 internal/gc/dryrun_cap_test.go diff --git a/internal/gc/blastcap_test.go b/internal/gc/blastcap_test.go index 437b690..c7cc3e5 100644 --- a/internal/gc/blastcap_test.go +++ b/internal/gc/blastcap_test.go @@ -40,3 +40,17 @@ func TestApplyBlastCap_LeavesAPlanWithinTheCapAlone(t *testing.T) { assert.Len(t, plan.prune, 1) assert.False(t, report.Capped) } + +func TestApplyBlastCap_ZeroLeavesReindexAlone(t *testing.T) { + t.Parallel() + + rc := &Reconciler{BlastCap: 0} + plan := &repairPlan{reindex: []string{"a"}, tombstone: []string{"b"}} + report := &DriftReport{} + + rc.applyBlastCap(context.Background(), "www", plan, report) + + assert.Equal(t, []string{"a"}, plan.reindex, + "reindex re-adds a lost index row for bytes that already exist; it destroys nothing, so no ceiling "+ + "on destruction should suppress it") +} diff --git a/internal/gc/dryrun_cap_test.go b/internal/gc/dryrun_cap_test.go new file mode 100644 index 0000000..1f7e13a --- /dev/null +++ b/internal/gc/dryrun_cap_test.go @@ -0,0 +1,29 @@ +package gc + +import ( + "context" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func TestReconcileSite_DryRunReportsFullDriftRegardlessOfTheBlastCap(t *testing.T) { + lister := &fakeReconcileLister{keys: []string{ + "www/deploys/20260101-000000-aaaaaaa/index.html", + "www/deploys/20260102-000000-bbbbbbb/index.html", + "www/deploys/20260103-000000-ccccccc/index.html", + }} + store := &fakeReconcileStore{} + rc := newReconciler(lister, store, &fakeMover{}) + rc.BlastCap = 0 + rc.Now = func() time.Time { return time.Date(2027, 1, 1, 0, 0, 0, 0, time.UTC) } + + report, err := rc.ReconcileSite(context.Background(), "www", true) + require.NoError(t, err) + + assert.Len(t, report.OrphanTombstoned, 3, + "the nightly sweep runs with the same cap as repair; letting the cap empty the REPORT turns a "+ + "misconfigured ceiling into a false all-clear, which is worse than the drift it hides") +} diff --git a/internal/gc/reconcile.go b/internal/gc/reconcile.go index d62eff1..d3e6b90 100644 --- a/internal/gc/reconcile.go +++ b/internal/gc/reconcile.go @@ -83,15 +83,16 @@ func (rc *Reconciler) ReconcileSite(ctx context.Context, site string, dryRun boo site, len(snap.indexed), rc.SitePrefix(site)) } plan := rc.classify(ctx, site, snap, &report) - rc.applyBlastCap(ctx, site, &plan, &report) if dryRun { report.Reindexed = plan.reindex report.OrphanTombstoned = plan.tombstone report.PGPruned = plan.prune + rc.predictBlastCap(ctx, plan, &report) rc.logDone(ctx, report, true) return report, nil } + rc.applyBlastCap(ctx, site, &plan, &report) if rc.Locker == nil { return report, fmt.Errorf("reconcile %s: live run without site Locker (wiring bug)", site) @@ -210,6 +211,11 @@ func (rc *Reconciler) classify(ctx context.Context, site string, snap siteSnapsh return plan } +func (rc *Reconciler) predictBlastCap(ctx context.Context, plan repairPlan, report *DriftReport) { + capped := plan + rc.applyBlastCap(ctx, report.Site, &capped, report) +} + func (rc *Reconciler) applyBlastCap(ctx context.Context, site string, plan *repairPlan, report *DriftReport) { destructive := len(plan.tombstone) + len(plan.prune) if destructive == 0 || (rc.BlastCap > 0 && destructive <= rc.BlastCap) { diff --git a/internal/gc/reconcile_race_test.go b/internal/gc/reconcile_race_test.go index 436d14b..7ab7b7e 100644 --- a/internal/gc/reconcile_race_test.go +++ b/internal/gc/reconcile_race_test.go @@ -391,7 +391,10 @@ func TestReconcile_DryRunPredictsTheCappedPlan(t *testing.T) { require.NoError(t, err) assert.True(t, report.Capped, "a dry run must predict the cap the live run will hit") - assert.Equal(t, []string{oldest}, report.OrphanTombstoned) + assert.Equal(t, []string{oldest, newest}, report.OrphanTombstoned, + "the report names every drifted deploy and warns separately that a live run would be capped; "+ + "truncating the report to the cap makes the nightly sweep under-report drift, and at cap 0 it "+ + "would report a clean fleet") assert.Empty(t, store.tombstoned, "a dry run mutates nothing") } From b093460c73965cb6b01fdadaff415e8632f9b119 Mon Sep 17 00:00:00 2001 From: Mrugesh Mohapatra Date: Mon, 17 Aug 2026 10:13:22 +0530 Subject: [PATCH 18/41] fix(cli): name the failing subcommand on stderr --- cmd/artemis/main.go | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/cmd/artemis/main.go b/cmd/artemis/main.go index f4d114f..62b9d74 100644 --- a/cmd/artemis/main.go +++ b/cmd/artemis/main.go @@ -57,9 +57,15 @@ func dispatchSubcommand(ctx context.Context, out io.Writer, args []string) (bool return true, fmt.Errorf("%s takes no arguments, got %q: it always sweeps every registered site", driftReportCommand, strings.Join(args[1:], " ")) } - return true, runDriftReport(ctx, out) + if err := runDriftReport(ctx, out); err != nil { + return true, fmt.Errorf("drift report failed: %w", err) + } + return true, nil case reconcileCommand: - return true, runReconcileCLI(ctx, out, args[1:]) + if err := runReconcileCLI(ctx, out, args[1:]); err != nil { + return true, fmt.Errorf("reconcile failed: %w", err) + } + return true, nil default: return true, fmt.Errorf("unknown subcommand %q: expected %s or %s", args[0], driftReportCommand, reconcileCommand) From b534f7c1a6bfbea7508bc4152bf2aa8f001ce2c2 Mon Sep 17 00:00:00 2001 From: Mrugesh Mohapatra Date: Mon, 17 Aug 2026 10:32:15 +0530 Subject: [PATCH 19/41] docs: correct blast cap and new url env vars --- .env.example | 4 +++- docs/ONBOARDING.md | 2 +- docs/README.md | 4 +++- docs/design/0005-drift-at-source.md | 20 ++++++-------------- 4 files changed, 13 insertions(+), 17 deletions(-) diff --git a/.env.example b/.env.example index b352954..554490a 100644 --- a/.env.example +++ b/.env.example @@ -32,6 +32,8 @@ VALKEY_ADDR=localhost:6379 # ALIAS_PRODUCTION_KEY_FORMAT=/production # ALIAS_PREVIEW_KEY_FORMAT=/preview # DEPLOY_PREFIX_FORMAT=/deploys/-/ +# PUBLIC_URL_PRODUCTION_FORMAT=https://.freecode.camp +# PUBLIC_URL_PREVIEW_FORMAT=https://.preview.freecode.camp # UPLOAD_MAX_BYTES=104857600 # 100 MiB # LOG_LEVEL=info # debug | info | warn | error @@ -63,7 +65,7 @@ VALKEY_ADDR=localhost:6379 # CLEANUP_RETENTION_DAYS=7 # days before a superseded deploy is GC-eligible # CLEANUP_RECENT_KEEP=3 # newest N deploys per site always kept # CLEANUP_GRACE=72h # min deploy age before GC; must be >= JWT_TTL_SECONDS -# CLEANUP_BLAST_CAP=0 # max deletes per sweep; 0 disables the cap +# CLEANUP_BLAST_CAP=10 # max deletes per sweep, oldest first; 0 refuses every repair # CLEANUP_TRASH_PREFIX=_trash/ # R2 prefix for tombstoned objects # CLEANUP_RECOVERY_DAYS=7 # days a tombstone survives before hard purge # CLEANUP_DRY_RUN= # 1/true: plan-only, execute nothing diff --git a/docs/ONBOARDING.md b/docs/ONBOARDING.md index 2e4a88c..fc9394e 100644 --- a/docs/ONBOARDING.md +++ b/docs/ONBOARDING.md @@ -156,7 +156,7 @@ Safety rails on the repair path: the site lock, a **second read inside the lock* Every reaping path records the tombstone row **before** it moves the bytes — `gc-site` at `internal/gc/gcsite.go:135` then `:138`, `reconcile` at `internal/gc/reconcile.go:286` then `:290`. -That order is forced by the purge being row-driven. Bytes moved into `_trash/` without a tombstone are invisible to `tombstone-purge` (which walks `tombstones`), invisible to the index, and invisible to `reconcile` (which lists the *site* prefix, not `_trash/`) — a permanent, undetectable leak. The inverse failure is benign: a row with its bytes still at the deploy prefix shows up as ordinary reindex drift, which the nightly sweep reports and `reconcile` repairs. Both paths log `tombstone_move_deferred` when they land in that state. +That order is forced by the purge being row-driven. Bytes moved into `_trash/` without a tombstone are invisible to `tombstone-purge` (which walks `tombstones`), invisible to the index, and invisible to `reconcile` (which lists the *site* prefix, not `_trash/`) — a permanent, undetectable leak. The inverse failure is bounded: a row with its bytes still at the deploy prefix shows up as reindex drift, which the nightly sweep reports. `reconcile` cannot repair it immediately — `ReindexDeploy` refuses while a tombstone for that id stands (`internal/pg/repo.go:46`) — so the bytes clear once `tombstone-purge` drops the row after `CLEANUP_RECOVERY_DAYS`. Both paths log `tombstone_move_deferred` when they land in that state. `gc-site` carried the leaky order until the drift-at-source sprint; if you find a doc or comment claiming otherwise, it predates that change. diff --git a/docs/README.md b/docs/README.md index 4afd081..0bf2264 100644 --- a/docs/README.md +++ b/docs/README.md @@ -93,6 +93,8 @@ Loaded + validated in `internal/config/config.go` (`Load()` — fails fast on th | `ALIAS_PRODUCTION_KEY_FORMAT` | `/production` | R2 alias key for production env | | `ALIAS_PREVIEW_KEY_FORMAT` | `/preview` | R2 alias key for preview env | | `DEPLOY_PREFIX_FORMAT` | `/deploys/-/` | R2 prefix per immutable deploy; must contain `` and `-` | +| `PUBLIC_URL_PRODUCTION_FORMAT` | `https://.freecode.camp` | URL returned to the CLI on a production finalize; must contain `` or boot fails | +| `PUBLIC_URL_PREVIEW_FORMAT` | `https://.preview.freecode.camp` | URL returned to the CLI on a preview finalize; must contain `` or boot fails | **Repo-creation (Apollo-11, feature-gated)** @@ -129,7 +131,7 @@ Loaded + validated in `internal/config/config.go` (`Load()` — fails fast on th | `CLEANUP_RETENTION_DAYS` | `7` | Days before a superseded deploy becomes GC-eligible | | `CLEANUP_RECENT_KEEP` | `3` | Newest N deploys per site kept regardless of age (rollback floor) | | `CLEANUP_GRACE` | `72h` | Minimum deploy age before GC; must be ≥ `JWT_TTL_SECONDS` and ≥ the 15s serve-cache TTL | -| `CLEANUP_BLAST_CAP` | `0` (disabled) | Max deploys reclaimed per sweep; an over-cap sweep reaps only the oldest N this run | +| `CLEANUP_BLAST_CAP` | `10` | Max deploys reclaimed per sweep, oldest first; `0` refuses every destructive repair | | `CLEANUP_TRASH_PREFIX` | `_trash/` | R2 prefix soft-deleted (tombstoned) objects move to before hard purge | | `CLEANUP_RECOVERY_DAYS` | `7` | Days a tombstone survives before the purge pass hard-deletes it | | `CLEANUP_DRY_RUN` | `false` | Plan-only GC: compute + log the delete set, execute nothing | diff --git a/docs/design/0005-drift-at-source.md b/docs/design/0005-drift-at-source.md index 6febc35..0b53b98 100644 --- a/docs/design/0005-drift-at-source.md +++ b/docs/design/0005-drift-at-source.md @@ -196,25 +196,17 @@ Everything verified during this audit, with a decision against each. "Accept, do | 11 | `outbox` has no retention — unbounded growth | **Backlog.** Small table, slow growth, no correctness impact. Needs a purge job eventually; not part of this wave. | | 12 | Dead worker code paths | **Backlog**, cosmetic. | | 13 | `RequireScope` / latched rate limiter behaviours | **Accept, documented.** Both behave as designed; the surprise is documentation, not code. Already captured in ONBOARDING §10. | +| 16 | `PlanSite` appended `in.Expired` (mtime ASC, `internal/pg/pending.go:29`) onto `Retain`'s output (mtime DESC, `internal/gc/retain.go:33-37`) without re-sorting, while the blast cap truncates from the tail (`internal/gc/plan.go`). Over-cap runs therefore reaped the **newest** abandoned sessions and starved retention entirely, while the reason string claimed "reaping oldest". Introduced by this sprint; found by the adversarial review, which reproduced it with a probe. | **Fixed** — merged set sorted newest-first before the cap; the test now asserts *which* deploys survive, not how many. | +| 17 | The blast cap ran before the dry-run branch in `ReconcileSite`, so a cap of 0 emptied the drift **report**. `drift-detect` runs dry with the same config, so a misconfigured ceiling would have reported a clean fleet. Introduced by this sprint. | **Fixed** — cap applies only to live runs; the dry run reports full drift and sets `Capped`/`CapReason` as a warning. | +| 18 | `newLiveAliasReader` validated the site *segment* but not the tail, so a format like `.freecode.camp/aliases-/production` passed boot and then fetched a key containing a literal `` — the same 404-for-every-site class P0-2 closed. | **Fixed** — boot refuses a `` token after the site segment. | | 15 | Every drift verdict op (`drift.selfcheck`, `drift.unreadable`, `drift.aliased_missing`) is absent from `cronShapedOps` (`internal/observability/sentry.go:377`), so `alertOnDrift`'s `captureBackground(v.Op, ...)` falls through to the transient-rate tracker: threshold 3 with a 26h reset window means a nightly alert is swallowed for two nights, and the in-memory counter resets on every pod restart, so with 3 replicas it may never escalate. **A live site serving nothing could page nobody.** | **Fixed** — all four verdict ops added, pinned by a test that fails if a new one is missed. | | 14 | `PublicURLForSite` (`internal/handler/handler.go:152`) is never assigned outside tests, so the hardcoded fallback at `deploy.go:377-382` always runs — the public URL returned to the CLI bakes `freecode.camp` and `.preview.` into the binary, while every other domain fact comes from config. There is no `ROOT_DOMAIN` setting. | **Fix with P0** (one-liner): derive the URL from the configured alias formats, or add the root domain to config. Cosmetic today, silently wrong the day the root domain or preview label changes. | ## Sequencing -Shipped together on `fix/artemis-drift-at-source` at the operator's direction — one sprint covering every bug, ten commits, each RED-first: - -| commit | phase | -| --- | --- | -| `fix(gc): read live aliases in the sweep keyspace` | P0-1 + P0-2 | -| `fix(cli): reject unknown subcommands and stray args` | P0, finding 9 | -| `fix(gc): record the tombstone row before moving bytes` | P0-4 | -| `fix(gc): make a zero blast cap refuse, not unleash` | P0-3 | -| `fix(handler): build public URLs from config` | finding 14 | -| `feat(pg): record a pending row when a deploy starts` | P2 | -| `feat(handler): register the deploy session at init` | P2 | -| `feat(gc): expire abandoned deploy sessions` | P2 | -| `fix(drift): alert on accruing reclaimable drift` | P3 + finding 15 | -| `test(gc): pin both key renderers to one layout` | P1, partial | +Shipped together on `fix/artemis-drift-at-source` at the operator's direction — one sprint covering every bug, each change RED-first. Read the branch log rather than a table here; an earlier revision of this section pinned a commit count that a later round of review fixes immediately invalidated. + +Phases map to commits by subject: `fix(gc): read live aliases...` is P0-1 + P0-2, `fix(cli): reject unknown subcommands...` is finding 9, `fix(gc): record the tombstone row...` is P0-4, `fix(gc): make a zero blast cap refuse...` is P0-3, `fix(handler): build public URLs...` is finding 14, the three `feat(...)` commits are P2, and `fix(drift): alert on accruing reclaimable drift` is P3 + finding 15. The remaining `fix`/`test` commits are review follow-ups, including three defects the sprint introduced into its own work: the blast cap silencing the drift report, the unsorted merge of expired-pending into the delete set, and a false claim that reconcile could repair bytes stranded by a failed tombstone-move. **P1 is deferred, deliberately.** The `Slug`/`Dirname` type split is prevention, not a bug, and a codebase-wide mechanical refactor landing alongside a behaviour change on the deploy hot path would make the review surface unreadable. What ships instead is the cheap structural half: boot validation that the alias formats and the deploy prefix share a site segment (so the dirname reconstruction is correct by construction, not by convention), plus a test that cross-checks all three rendered keys — site, deploy, trash — between the two renderers under the production FQDN format. The type split remains the right next wave. From 04f48cf7b26556bc6f1fd76ad16a5e3584cbde1d Mon Sep 17 00:00:00 2001 From: Mrugesh Mohapatra Date: Mon, 17 Aug 2026 10:32:15 +0530 Subject: [PATCH 20/41] fix(gc): reap the oldest across both delete sources --- internal/gc/errorpath_test.go | 6 ++++-- internal/gc/gcsite.go | 4 ++-- internal/gc/plan.go | 15 ++++++++++++++- internal/gc/plan_test.go | 28 ++++++++++++++++++++++++++++ internal/gc/reconcile.go | 8 ++------ 5 files changed, 50 insertions(+), 11 deletions(-) diff --git a/internal/gc/errorpath_test.go b/internal/gc/errorpath_test.go index 8262dd9..d44f21d 100644 --- a/internal/gc/errorpath_test.go +++ b/internal/gc/errorpath_test.go @@ -125,8 +125,10 @@ func TestGC_MoveFailureLeavesTheTombstoneRowForTheNextRun(t *testing.T) { require.ErrorContains(t, err, "tombstone-move") assert.Equal(t, []string{"www/d-old"}, store.tombstoned, - "the row landed before the move, so the bytes stay at the deploy prefix and surface as reindex "+ - "drift — visible to drift-detect and repairable by reconcile, unlike bytes stranded in _trash/") + "the row landed before the move, so the bytes stay at the deploy prefix where the sweep still "+ + "reports them; reindex is refused while the tombstone stands (repo.go:46), so they clear only "+ + "after tombstone-purge drops the row at CLEANUP_RECOVERY_DAYS — bounded and visible, unlike "+ + "bytes moved to _trash/ with no row, which nothing lists at all") assert.Empty(t, res.Tombstoned, "a deploy whose bytes never moved is not reported as reclaimed") require.Len(t, mover.moves, 1, "aborts on the first failed move, never proceeding to the next deploy") } diff --git a/internal/gc/gcsite.go b/internal/gc/gcsite.go index 512b670..522aa3e 100644 --- a/internal/gc/gcsite.go +++ b/internal/gc/gcsite.go @@ -138,8 +138,8 @@ func (g *SiteGC) Run(ctx context.Context, site string, dryRun bool) (GCResult, e if _, err := g.Mover.MovePrefix(opCtx, src, dst); err != nil { slog.WarnContext(opCtx, "gc.site.tombstone_move_deferred", "site", site, "deploy_id", d.ID, "trash_prefix", dst, "err", err, - "detail", "the row landed before the move, so the bytes stay at the deploy prefix and "+ - "surface as reindex drift for the next drift sweep") + "detail", "the row landed before the move, so the bytes stay at the deploy prefix; the "+ + "tombstone blocks reindex until tombstone-purge clears it after the recovery window") return fmt.Errorf("tombstone-move %s: %w", d.ID, err) } res.Tombstoned = append(res.Tombstoned, d.ID) diff --git a/internal/gc/plan.go b/internal/gc/plan.go index 8de1b8c..a78791b 100644 --- a/internal/gc/plan.go +++ b/internal/gc/plan.go @@ -1,6 +1,9 @@ package gc -import "fmt" +import ( + "fmt" + "sort" +) type Plan struct { Site string @@ -10,9 +13,19 @@ type Plan struct { Reason string } +func sortNewestFirst(ds []Deploy) { + sort.SliceStable(ds, func(i, j int) bool { + if !ds[i].Mtime.Equal(ds[j].Mtime) { + return ds[i].Mtime.After(ds[j].Mtime) + } + return ds[i].ID > ds[j].ID + }) +} + func PlanSite(site string, in RetainInput, p Policy, blastCap int) Plan { _, del := Retain(in, p) del = append(del, in.Expired...) + sortNewestFirst(del) plan := Plan{Site: site} if len(del) > 0 && blastCap <= 0 { diff --git a/internal/gc/plan_test.go b/internal/gc/plan_test.go index 390e4c2..30dd9a4 100644 --- a/internal/gc/plan_test.go +++ b/internal/gc/plan_test.go @@ -5,6 +5,7 @@ import ( "time" "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" ) func oldDeploys(n int, eachBytes int64) []Deploy { @@ -56,6 +57,14 @@ func TestPlanSite_BlastCapZeroRefusesEveryDelete(t *testing.T) { assert.Contains(t, plan.Reason, "blast-cap 0") } +func planIDs(p Plan) []string { + out := make([]string, 0, len(p.Delete)) + for _, d := range p.Delete { + out = append(out, d.ID) + } + return out +} + func TestPlanSite_ExpiredPendingJoinsTheDeleteSetUnderTheSameCap(t *testing.T) { in := RetainInput{ Deploys: oldDeploys(6, 100), @@ -69,3 +78,22 @@ func TestPlanSite_ExpiredPendingJoinsTheDeleteSetUnderTheSameCap(t *testing.T) { assert.Len(t, plan.Delete, 2, "abandoned sessions must be bounded by the same ceiling as retention, not appended past it") } + +func TestPlanSite_CapReapsTheOldestAcrossBothDeleteSources(t *testing.T) { + in := RetainInput{ + Deploys: oldDeploys(4, 100), + Expired: []Deploy{ + {ID: "pend-newest", Mtime: testNow.Add(-73 * time.Hour)}, + {ID: "pend-oldest", Mtime: testNow.Add(-500 * time.Hour)}, + }, + Now: testNow, + } + + plan := PlanSite("www", in, testPolicy(), 2) + + require.True(t, plan.Aborted) + assert.Equal(t, []string{"pend-oldest", "d-old"}, planIDs(plan), + "Retain returns newest-first while ExpiredPendingDeploys returns oldest-first, and the cap slices "+ + "the tail; concatenating them unsorted makes the tail the NEWEST abandoned sessions while the "+ + "reason string still claims it reaped the oldest, starving retention on any site over the cap") +} diff --git a/internal/gc/reconcile.go b/internal/gc/reconcile.go index d3e6b90..82658e8 100644 --- a/internal/gc/reconcile.go +++ b/internal/gc/reconcile.go @@ -88,7 +88,8 @@ func (rc *Reconciler) ReconcileSite(ctx context.Context, site string, dryRun boo report.Reindexed = plan.reindex report.OrphanTombstoned = plan.tombstone report.PGPruned = plan.prune - rc.predictBlastCap(ctx, plan, &report) + capped := plan + rc.applyBlastCap(ctx, site, &capped, &report) rc.logDone(ctx, report, true) return report, nil } @@ -211,11 +212,6 @@ func (rc *Reconciler) classify(ctx context.Context, site string, snap siteSnapsh return plan } -func (rc *Reconciler) predictBlastCap(ctx context.Context, plan repairPlan, report *DriftReport) { - capped := plan - rc.applyBlastCap(ctx, report.Site, &capped, report) -} - func (rc *Reconciler) applyBlastCap(ctx context.Context, site string, plan *repairPlan, report *DriftReport) { destructive := len(plan.tombstone) + len(plan.prune) if destructive == 0 || (rc.BlastCap > 0 && destructive <= rc.BlastCap) { From 9f90a066e30881efed8c9bee315768c9f975645a Mon Sep 17 00:00:00 2001 From: Mrugesh Mohapatra Date: Mon, 17 Aug 2026 10:32:15 +0530 Subject: [PATCH 21/41] fix(gc): refuse a site token outside the site segment --- cmd/artemis/driftalert.go | 32 +++++++++++++++++++++--- cmd/artemis/driftalert_threshold_test.go | 2 ++ cmd/artemis/gcwire.go | 9 ++++++- cmd/artemis/gcwire_test.go | 8 ++++++ 4 files changed, 47 insertions(+), 4 deletions(-) diff --git a/cmd/artemis/driftalert.go b/cmd/artemis/driftalert.go index e41f2b2..5e64df9 100644 --- a/cmd/artemis/driftalert.go +++ b/cmd/artemis/driftalert.go @@ -5,6 +5,7 @@ import ( "errors" "fmt" "log/slog" + "sort" "strings" ) @@ -42,12 +43,13 @@ func classifyDrift(res sweepResult) driftVerdict { return driftVerdict{Op: opDriftUnreadable, Err: unread, Fails: true} } if reindex, tombstone, _, _ := res.totals(); reindex+tombstone >= reclaimableAlertThreshold { + sites := reclaimableSites(res.Reports) return driftVerdict{ Op: opDriftReclaimable, Err: fmt.Errorf( - "%d deploys are reclaimable across %d sites (>= %d): storage is accruing faster than it is "+ - "collected; run `artemis reconcile --apply` and find what stopped expiring", - reindex+tombstone, res.Stats.Sites, reclaimableAlertThreshold), + "%d deploys are reclaimable across %s (>= %d): storage is accruing faster than it is "+ + "collected; run `artemis reconcile --apply` for each and find what stopped expiring", + reindex+tombstone, strings.Join(sites, ", "), reclaimableAlertThreshold), } } return driftVerdict{} @@ -61,6 +63,30 @@ func unreadableErr(unreadable []string, sites int) error { len(unreadable), sites, strings.Join(unreadable, ", ")) } +func reclaimableSites(reports []siteDrift) []string { + type sited struct { + site string + n int + } + var ranked []sited + for _, r := range reports { + if n := len(r.Reindex) + len(r.Tombstone); n > 0 { + ranked = append(ranked, sited{r.Site, n}) + } + } + sort.Slice(ranked, func(i, j int) bool { + if ranked[i].n != ranked[j].n { + return ranked[i].n > ranked[j].n + } + return ranked[i].site < ranked[j].site + }) + out := make([]string, 0, len(ranked)) + for _, s := range ranked { + out = append(out, fmt.Sprintf("%s (%d)", s.site, s.n)) + } + return out +} + func unreadableSites(reports []siteDrift) []string { var out []string for _, r := range reports { diff --git a/cmd/artemis/driftalert_threshold_test.go b/cmd/artemis/driftalert_threshold_test.go index 4642a1f..b67b28f 100644 --- a/cmd/artemis/driftalert_threshold_test.go +++ b/cmd/artemis/driftalert_threshold_test.go @@ -41,6 +41,8 @@ func TestClassifyDrift_AlertsOnceReclaimableDriftAccumulates(t *testing.T) { assert.False(t, v.Fails, "reclaimable drift is storage cost, not an outage: alert, do not fail the run") require.Error(t, v.Err) assert.Contains(t, v.Err.Error(), "artemis reconcile") + assert.Contains(t, v.Err.Error(), "www.freecode.camp", + "a fleet-wide count with a literal placeholder tells the operator nothing about where to look") } func TestClassifyDrift_AliasedMissingOutranksTheReclaimableThreshold(t *testing.T) { diff --git a/cmd/artemis/gcwire.go b/cmd/artemis/gcwire.go index b9ee844..2097ff6 100644 --- a/cmd/artemis/gcwire.go +++ b/cmd/artemis/gcwire.go @@ -160,7 +160,14 @@ func newLiveAliasReader(getter aliasGetter, deployFormat string, formats ...stri "key under a different site segment is unreachable and would 404 for every site", f, seg, deployFormat, deploySeg) } - tails = append(tails, f[len(seg)+1:]) + tail := f[len(seg)+1:] + if strings.Contains(tail, "") { + return nil, fmt.Errorf( + "alias key format %q keeps a token after its site segment: only the segment is "+ + "rendered from the dirname, so the rest is fetched literally and 404s for every site", + f) + } + tails = append(tails, tail) } return func(ctx context.Context, dirname string) (map[string]struct{}, error) { out := map[string]struct{}{} diff --git a/cmd/artemis/gcwire_test.go b/cmd/artemis/gcwire_test.go index 542d540..663db99 100644 --- a/cmd/artemis/gcwire_test.go +++ b/cmd/artemis/gcwire_test.go @@ -285,3 +285,11 @@ func TestGCPolicyFromConfig(t *testing.T) { assert.Equal(t, 7*24*time.Hour, p.Retention) assert.Equal(t, 15*time.Second, p.ServeCacheTTL) } + +func TestNewLiveAliasReader_RejectsASiteTokenOutsideTheSiteSegment(t *testing.T) { + _, err := newLiveAliasReader(&recordingAliasGetter{}, domainFormat, + ".freecode.camp/aliases-/production") + require.Error(t, err, + "the reader substitutes nothing after the site segment, so a surviving is fetched literally "+ + "and 404s for every site — the same silent-inert failure this constructor exists to refuse") +} From 5fa53179df8df1d9c64ddd8b8abeecbc9c72a387 Mon Sep 17 00:00:00 2001 From: Mrugesh Mohapatra Date: Mon, 17 Aug 2026 17:42:44 +0530 Subject: [PATCH 22/41] fix(handler): record tombstones before moving bytes --- internal/handler/deploy_delete.go | 8 +- internal/handler/destructive_ordering_test.go | 134 ++++++++++++++++++ internal/handler/site_purge_test.go | 9 +- internal/handler/site_register.go | 8 +- 4 files changed, 149 insertions(+), 10 deletions(-) create mode 100644 internal/handler/destructive_ordering_test.go diff --git a/internal/handler/deploy_delete.go b/internal/handler/deploy_delete.go index 29e11b7..4450aa3 100644 --- a/internal/handler/deploy_delete.go +++ b/internal/handler/deploy_delete.go @@ -63,16 +63,16 @@ func (h *Handlers) SiteDeployDelete(w http.ResponseWriter, r *http.Request) { if bytesErr != nil { deployBytes = 0 } + if err := h.Tombstones.RecordTombstone(opCtx, h.DeployPrefix.SiteDirname(site), deployID, deployBytes); err != nil { + writeUpstreamError(w, r, http.StatusBadGateway, "tombstone_record_failed", "pg.tombstone.record", err) + return nil + } var err error moved, err = h.R2.MovePrefix(opCtx, h.deployPrefix(site, deployID), h.trashPrefix(site, deployID)) if err != nil { writeUpstreamError(w, r, http.StatusBadGateway, "r2_move_failed", "r2.move.tombstone", err) return nil } - if err := h.Tombstones.RecordTombstone(opCtx, h.DeployPrefix.SiteDirname(site), deployID, deployBytes); err != nil { - writeUpstreamError(w, r, http.StatusBadGateway, "tombstone_record_failed", "pg.tombstone.record", err) - return nil - } success = true return nil diff --git a/internal/handler/destructive_ordering_test.go b/internal/handler/destructive_ordering_test.go new file mode 100644 index 0000000..04f292d --- /dev/null +++ b/internal/handler/destructive_ordering_test.go @@ -0,0 +1,134 @@ +package handler + +import ( + "context" + "encoding/json" + "errors" + "net/http" + "net/http/httptest" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +type orderLog struct{ ops []string } + +type orderR2 struct { + *fakeR2 + log *orderLog +} + +func (o *orderR2) MovePrefix(ctx context.Context, src, dst string) (int, error) { + o.log.ops = append(o.log.ops, "bytes") + return o.fakeR2.MovePrefix(ctx, src, dst) +} + +type orderTombstones struct { + fakeTombstones + log *orderLog +} + +func (o *orderTombstones) RecordSitePurge(ctx context.Context, site string) error { + o.log.ops = append(o.log.ops, "row") + return o.fakeTombstones.RecordSitePurge(ctx, site) +} + +func (o *orderTombstones) RecordTombstone(ctx context.Context, site, id string, bytes int64) error { + o.log.ops = append(o.log.ops, "row") + return o.fakeTombstones.RecordTombstone(ctx, site, id, bytes) +} + +func registerExample(t *testing.T, h *Handlers) { + t.Helper() + regBody, _ := json.Marshal(SiteRegisterRequest{Slug: "example", Teams: []string{"staff"}}) + require.Equal(t, http.StatusCreated, callRegister(h, regBody, "alice", "tok").Code) +} + +func callPurge(h *Handlers) *httptest.ResponseRecorder { + return withChiRoute(http.MethodDelete, "/api/site/{slug}", + "/api/site/example?purge=true", nil, + map[string]string{}, + h.SiteDelete, + contextWithLogin(context.Background(), "alice", "tok"), + ) +} + +func TestSitePurge_RecordsTheSiteTombstoneBeforeMovingBytes(t *testing.T) { + log := &orderLog{} + store := &orderR2{fakeR2: newFakeR2(), log: log} + store.objects["example/deploys/20260420-141522-abc1234/index.html"] = []byte("hi") + + h, _ := newTestHandlers(t, staffCallerGH(), standardSites(), store) + tomb := &orderTombstones{log: log} + h.Tombstones = tomb + registerExample(t, h) + + w := callPurge(h) + require.Equal(t, http.StatusOK, w.Code, w.Body.String()) + + require.NotEmpty(t, log.ops) + assert.Equal(t, "row", log.ops[0], + "a crash between the two writes must leave a site tombstone naming bytes still in place — never a "+ + "whole site in _trash/ that no tombstone dates, which tombstone-purge, the index and reconcile "+ + "all fail to list, forever") +} + +func TestSitePurge_LeavesBytesInPlaceWhenTheRowWriteFails(t *testing.T) { + store := newFakeR2() + store.objects["example/deploys/20260420-141522-abc1234/index.html"] = []byte("hi") + + h, _ := newTestHandlers(t, staffCallerGH(), standardSites(), store) + h.Tombstones = &fakeTombstones{err: errors.New("pg down")} + registerExample(t, h) + + w := callPurge(h) + require.Equal(t, http.StatusBadGateway, w.Code, w.Body.String()) + assert.Contains(t, w.Body.String(), "tombstone_record_failed") + + store.mu.Lock() + defer store.mu.Unlock() + _, live := store.objects["example/deploys/20260420-141522-abc1234/index.html"] + assert.True(t, live, + "bytes must not move once the row that would date them is known to have failed; the site stays "+ + "registered and the purge is retryable") +} + +func TestSiteDeployDelete_RecordsTheTombstoneBeforeMovingBytes(t *testing.T) { + deployID := "20260420-141522-abc1234" + log := &orderLog{} + store := &orderR2{fakeR2: newFakeR2(), log: log} + store.objects["www/deploys/"+deployID+"/index.html"] = []byte("hi") + + h, _ := newTestHandlers(t, authedGH(), standardSites(), store) + tomb := &orderTombstones{log: log} + h.Tombstones = tomb + + w := callDeployDelete(h, "www", deployID) + require.Equal(t, http.StatusOK, w.Code, w.Body.String()) + + require.NotEmpty(t, log.ops) + assert.Equal(t, "row", log.ops[0], + "same rule as gc-site and reconcile: the tombstone row lands first, so a crash strands bytes at the "+ + "deploy prefix where the drift sweep reports them, not in _trash/ where nothing lists them") +} + +func TestSiteDeployDelete_LeavesBytesInPlaceWhenTheRowWriteFails(t *testing.T) { + deployID := "20260420-141522-abc1234" + store := newFakeR2() + store.objects["www/deploys/"+deployID+"/index.html"] = []byte("hi") + + h, _ := newTestHandlers(t, authedGH(), standardSites(), store) + h.Tombstones = &fakeTombstones{err: errors.New("pg down")} + + w := callDeployDelete(h, "www", deployID) + require.Equal(t, http.StatusBadGateway, w.Code, w.Body.String()) + assert.Contains(t, w.Body.String(), "tombstone_record_failed") + + store.mu.Lock() + defer store.mu.Unlock() + _, live := store.objects["www/deploys/"+deployID+"/index.html"] + assert.True(t, live, + "a deploy whose tombstone could not be written keeps serving from its prefix and the delete is "+ + "retryable; moving it first would orphan it in _trash/ with no row to date it") +} diff --git a/internal/handler/site_purge_test.go b/internal/handler/site_purge_test.go index 8cae219..f01f3e0 100644 --- a/internal/handler/site_purge_test.go +++ b/internal/handler/site_purge_test.go @@ -117,7 +117,11 @@ func TestSitePurge_FailedMoveKeepsSiteRetryable(t *testing.T) { slugs[i] = r.Slug } assert.Contains(t, slugs, "example", "failed purge must not deregister the site (still retryable)") - assert.Empty(t, tomb.recorded, "no tombstone written when the move failed") + assert.Equal(t, []string{"example"}, tomb.purged, + "the site tombstone lands before the move, so a failed move leaves the row naming the bytes still "+ + "in place; the retry re-records it, restarting the recovery clock exactly as "+ + "TestRepo_RecordSitePurge_RestartsTheRecoveryWindow pins on the repo side") + assert.Empty(t, tomb.recorded, "no per-deploy tombstone rows from a site purge") retryW := withChiRoute(http.MethodDelete, "/api/site/{slug}", "/api/site/example?purge=true", nil, @@ -133,7 +137,8 @@ func TestSitePurge_FailedMoveKeepsSiteRetryable(t *testing.T) { assert.Truef(t, hasPrefix(k, "_trash/example/"), "retry cascaded every example/ object into _trash, found %q live", k) } store.mu.Unlock() - assert.Equal(t, []string{"example"}, tomb.purged, "retry records the site purge") + assert.Equal(t, []string{"example", "example"}, tomb.purged, + "each purge attempt records the site tombstone before moving; the second entry is the retry") gone := callSitesList(h, "alice", "tok") require.Equal(t, http.StatusOK, gone.Code) diff --git a/internal/handler/site_register.go b/internal/handler/site_register.go index 79eeccf..888eea4 100644 --- a/internal/handler/site_register.go +++ b/internal/handler/site_register.go @@ -252,16 +252,16 @@ func (h *Handlers) SiteDelete(w http.ResponseWriter, r *http.Request) { success bool ) lockErr := h.withSiteLock(opCtx, dirname, func() error { + if err := h.Tombstones.RecordSitePurge(opCtx, dirname); err != nil { + writeUpstreamError(w, r, http.StatusBadGateway, "tombstone_record_failed", "pg.tombstone.site-purge", err) + return nil + } var err error moved, err = h.R2.MovePrefix(opCtx, dirname+"/", base+dirname+"/") if err != nil { writeUpstreamError(w, r, http.StatusBadGateway, "r2_move_failed", "r2.move.site-purge", err) return nil } - if err := h.Tombstones.RecordSitePurge(opCtx, dirname); err != nil { - writeUpstreamError(w, r, http.StatusBadGateway, "tombstone_record_failed", "pg.tombstone.site-purge", err) - return nil - } if err := h.Registry.Delete(opCtx, slug); err != nil { writeRegistryDeleteError(w, r, err) return nil From 030a9c6364df7049ab3475342ff7f1ca082942ad Mon Sep 17 00:00:00 2001 From: Mrugesh Mohapatra Date: Mon, 17 Aug 2026 17:44:07 +0530 Subject: [PATCH 23/41] fix(auth): give the scope guard an honest signature --- internal/auth/jwt.go | 14 +++++-------- internal/auth/jwt_test.go | 9 ++++---- internal/handler/deploy.go | 4 ++-- internal/handler/destructive_ordering_test.go | 21 +++++++++++++++++++ 4 files changed, 33 insertions(+), 15 deletions(-) diff --git a/internal/auth/jwt.go b/internal/auth/jwt.go index da4f2d5..c89f33e 100644 --- a/internal/auth/jwt.go +++ b/internal/auth/jwt.go @@ -37,15 +37,11 @@ type DeploySessionClaims struct { jwt.RegisteredClaims } -// RequireScope verifies that the JWT was issued for exactly this -// (login, site, deployId) triple. Returns an error otherwise. -func (c DeploySessionClaims) RequireScope(login, site, deployID string) error { - if c.Subject != login { - return fmt.Errorf("auth: jwt sub %q != expected %q", c.Subject, login) - } - if c.Site != site { - return fmt.Errorf("auth: jwt site %q != expected %q", c.Site, site) - } +// RequireDeployID verifies the JWT was minted for exactly this deploy. +// It is the only request-supplied value to check: the upload/finalize +// write target is rendered from c.Site and the subject is bound by the +// signature, so neither has an independent request-side counterpart. +func (c DeploySessionClaims) RequireDeployID(deployID string) error { if c.DeployID != deployID { return fmt.Errorf("auth: jwt deployId %q != expected %q", c.DeployID, deployID) } diff --git a/internal/auth/jwt_test.go b/internal/auth/jwt_test.go index c7489cb..abb9eb9 100644 --- a/internal/auth/jwt_test.go +++ b/internal/auth/jwt_test.go @@ -120,10 +120,11 @@ func TestRequireScope_RejectsWrongDeployID(t *testing.T) { claims, err := s.Verify(tok) require.NoError(t, err) - require.NoError(t, claims.RequireScope("alice", "www", "d-1")) - require.Error(t, claims.RequireScope("alice", "www", "d-2")) - require.Error(t, claims.RequireScope("alice", "learn", "d-1")) - require.Error(t, claims.RequireScope("bob", "www", "d-1")) + require.NoError(t, claims.RequireDeployID("d-1")) + require.Error(t, claims.RequireDeployID("d-2"), + "the old three-argument RequireScope was called with its own claims as the expected login and "+ + "site, so those two checks could never fire; the honest signature checks the one value that "+ + "actually arrives from the request") } func TestNewSigner_RejectsShortKey(t *testing.T) { diff --git a/internal/handler/deploy.go b/internal/handler/deploy.go index 5fe54d8..427d901 100644 --- a/internal/handler/deploy.go +++ b/internal/handler/deploy.go @@ -108,7 +108,7 @@ func (h *Handlers) DeployUpload(w http.ResponseWriter, r *http.Request) { return } deployID := chi.URLParam(r, "deployId") - if err := claims.RequireScope(claims.Subject, claims.Site, deployID); err != nil { + if err := claims.RequireDeployID(deployID); err != nil { writeError(w, http.StatusForbidden, "jwt_wrong_deploy", "deploy-session jwt does not match url deploy id") return } @@ -191,7 +191,7 @@ func (h *Handlers) DeployFinalize(w http.ResponseWriter, r *http.Request) { return } deployID := chi.URLParam(r, "deployId") - if err := claims.RequireScope(claims.Subject, claims.Site, deployID); err != nil { + if err := claims.RequireDeployID(deployID); err != nil { writeError(w, http.StatusForbidden, "jwt_wrong_deploy", "deploy-session jwt does not match url deploy id") return } diff --git a/internal/handler/destructive_ordering_test.go b/internal/handler/destructive_ordering_test.go index 04f292d..8e5a267 100644 --- a/internal/handler/destructive_ordering_test.go +++ b/internal/handler/destructive_ordering_test.go @@ -132,3 +132,24 @@ func TestSiteDeployDelete_LeavesBytesInPlaceWhenTheRowWriteFails(t *testing.T) { "a deploy whose tombstone could not be written keeps serving from its prefix and the delete is "+ "retryable; moving it first would orphan it in _trash/ with no row to date it") } + +func TestDeployFinalize_RejectsWrongDeployID(t *testing.T) { + h, jwt := newTestHandlers(t, &fakeGH{}, standardSites(), newFakeR2()) + + tok, _, err := jwt.Sign("alice", "www", "20260420-141522-abc1234") + require.NoError(t, err) + + body, _ := json.Marshal(DeployFinalizeRequest{Mode: "preview", Files: []string{"index.html"}}) + w := withChiRoute(http.MethodPost, "/api/deploy/{deployId}/finalize", + "/api/deploy/wrong-deploy/finalize", + body, + map[string]string{"Authorization": "Bearer " + tok}, + h.RequireDeployJWT(http.HandlerFunc(h.DeployFinalize)).ServeHTTP, + context.Background(), + ) + + assert.Equal(t, http.StatusForbidden, w.Code, w.Body.String()) + assert.Contains(t, w.Body.String(), "jwt_wrong_deploy", + "the deployId comparison is the only live check in the scope guard; the site is pinned by "+ + "rendering the write target from claims.Site, and the subject by the signature itself") +} From a4c3abc7ca3b2c47a7b9ffe1107bc1152ef7f72f Mon Sep 17 00:00:00 2001 From: Mrugesh Mohapatra Date: Mon, 17 Aug 2026 17:45:01 +0530 Subject: [PATCH 24/41] fix(auth): classify secondary throttles as rate limits --- internal/auth/github.go | 15 +++-- .../auth/github_secondary_ratelimit_test.go | 65 +++++++++++++++++++ 2 files changed, 75 insertions(+), 5 deletions(-) create mode 100644 internal/auth/github_secondary_ratelimit_test.go diff --git a/internal/auth/github.go b/internal/auth/github.go index 7dc83fe..a0ab883 100644 --- a/internal/auth/github.go +++ b/internal/auth/github.go @@ -497,10 +497,15 @@ func IsGitHubUnavailable(err error) bool { return errors.Is(err, ErrGitHubUnavai // (non-rate-limited) response. func IsGitHubUnauthenticated(err error) bool { return errors.Is(err, ErrGitHubUnauthenticated) } -// isRateLimited reports whether resp is a GitHub primary-rate-limit -// response. Authoritative signal is the `X-RateLimit-Remaining: 0` -// header (RFC 6585 §4 + GitHub REST docs). Body-substring detection -// (pre-B16) was fragile against changes to GitHub's error wording. +// isRateLimited reports whether resp is a GitHub rate-limit response. +// Primary limits signal `X-RateLimit-Remaining: 0` (RFC 6585 §4 + +// GitHub REST docs). Secondary (abuse) limits return 403 with a +// Retry-After header while Remaining is typically non-zero; before +// this check they fell through to the plain-403 branch and were +// negative-cached as unauthenticated for up to negCacheCap. func isRateLimited(resp *http.Response) bool { - return resp.Header.Get("X-RateLimit-Remaining") == "0" + if resp.Header.Get("X-RateLimit-Remaining") == "0" { + return true + } + return resp.StatusCode == http.StatusForbidden && resp.Header.Get("Retry-After") != "" } diff --git a/internal/auth/github_secondary_ratelimit_test.go b/internal/auth/github_secondary_ratelimit_test.go new file mode 100644 index 0000000..1ac71c1 --- /dev/null +++ b/internal/auth/github_secondary_ratelimit_test.go @@ -0,0 +1,65 @@ +package auth + +import ( + "context" + "net/http" + "net/http/httptest" + "sync/atomic" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func secondaryLimitServer(t *testing.T, calls *atomic.Int32) *httptest.Server { + t.Helper() + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + calls.Add(1) + w.Header().Set("Retry-After", "60") + w.Header().Set("X-RateLimit-Remaining", "4999") + w.WriteHeader(http.StatusForbidden) + _, _ = w.Write([]byte(`{"message":"You have exceeded a secondary rate limit. Please wait a few minutes before you try again."}`)) + })) + t.Cleanup(srv.Close) + return srv +} + +func TestValidateToken_SecondaryRateLimitIsNotAnAuthFailure(t *testing.T) { + var calls atomic.Int32 + srv := secondaryLimitServer(t, &calls) + c := NewGitHubClient(GitHubClientConfig{APIBase: srv.URL}) + + _, err := c.ValidateToken(context.Background(), "tok-abc") + + require.Error(t, err) + assert.True(t, IsGitHubRateLimited(err), + "a secondary-limit 403 carries Retry-After with non-zero X-RateLimit-Remaining; classifying it as "+ + "unauthenticated tells the operator a working credential is bad") + assert.False(t, IsGitHubUnauthenticated(err)) +} + +func TestValidateToken_SecondaryRateLimitIsNeverNegativeCached(t *testing.T) { + var calls atomic.Int32 + srv := secondaryLimitServer(t, &calls) + c := NewGitHubClient(GitHubClientConfig{APIBase: srv.URL}) + + _, _ = c.ValidateToken(context.Background(), "tok-abc") + _, err := c.ValidateToken(context.Background(), "tok-abc") + + require.Error(t, err) + assert.EqualValues(t, 2, calls.Load(), + "the old path cached the throttle as ErrGitHubUnauthenticated for up to 30s, so every retry inside "+ + "that window got a 401 from cache without any upstream call; a throttle must stay uncached") + assert.True(t, IsGitHubRateLimited(err)) +} + +func TestIsTeamMember_SecondaryRateLimitSurfacesAsRateLimited(t *testing.T) { + var calls atomic.Int32 + srv := secondaryLimitServer(t, &calls) + c := NewGitHubClient(GitHubClientConfig{APIBase: srv.URL, Org: "freeCodeCamp"}) + + _, err := c.IsTeamMember(context.Background(), "tok", "alice", "team-eng") + + require.Error(t, err) + assert.True(t, IsGitHubRateLimited(err)) +} From bb61e8bc4e91ab9571d191429f3d16895031ebc0 Mon Sep 17 00:00:00 2001 From: Mrugesh Mohapatra Date: Mon, 17 Aug 2026 17:46:50 +0530 Subject: [PATCH 25/41] refactor(gc): one owner for oldest-first cap selection --- internal/gc/capoldest_test.go | 62 +++++++++++++++++++++++++++++++++++ internal/gc/plan.go | 30 ++++++++++++----- internal/gc/plan_test.go | 2 +- internal/gc/reconcile.go | 8 ++--- 4 files changed, 86 insertions(+), 16 deletions(-) create mode 100644 internal/gc/capoldest_test.go diff --git a/internal/gc/capoldest_test.go b/internal/gc/capoldest_test.go new file mode 100644 index 0000000..32bb53f --- /dev/null +++ b/internal/gc/capoldest_test.go @@ -0,0 +1,62 @@ +package gc + +import ( + "context" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func TestBlastCap_BothPathsSelectTheSameOldestSurvivors(t *testing.T) { + ids := []string{ + "20260101-000000-aaaaaaa", + "20260201-000000-bbbbbbb", + "20260301-000000-ccccccc", + "20260401-000000-ddddddd", + } + mt := func(id string) time.Time { + ts, err := time.Parse("20060102-150405", id[:15]) + require.NoError(t, err) + return ts + } + + deploys := make([]Deploy, 0, len(ids)) + for _, id := range ids { + deploys = append(deploys, Deploy{ID: id, Mtime: mt(id)}) + } + plan := PlanSite("www", RetainInput{Deploys: deploys, Now: mt(ids[3]).Add(365 * 24 * time.Hour)}, + Policy{RecentKeep: 0, Grace: time.Hour, Retention: time.Hour}, 2) + require.True(t, plan.Aborted) + + rc := &Reconciler{BlastCap: 2} + rp := &repairPlan{tombstone: append([]string(nil), ids...)} + rep := &DriftReport{} + rc.applyBlastCap(context.Background(), "www", rp, rep) + require.True(t, rep.Capped) + + assert.Equal(t, []string{ids[0], ids[1]}, planIDs(plan), + "retention GC and reconcile advertise the same contract — an over-cap run reaps the oldest N — "+ + "so given identical candidates they must pick identical survivors") + assert.Equal(t, []string{ids[0], ids[1]}, rp.tombstone, + "one path once sorted DESC and took the tail while the other sorted ASC and took the head; the "+ + "selection must live in exactly one helper so the conventions cannot diverge again") +} + +func TestApplyBlastCap_TombstonesConsumeTheBudgetBeforePrunes(t *testing.T) { + rc := &Reconciler{BlastCap: 3} + rp := &repairPlan{ + tombstone: []string{"20260101-000000-aaaaaaa", "20260201-000000-bbbbbbb"}, + prune: []string{"20260102-000000-xxxxxxx", "20260202-000000-yyyyyyy"}, + } + rep := &DriftReport{} + + rc.applyBlastCap(context.Background(), "www", rp, rep) + + require.True(t, rep.Capped) + assert.Equal(t, []string{"20260101-000000-aaaaaaa", "20260201-000000-bbbbbbb"}, rp.tombstone) + assert.Equal(t, []string{"20260102-000000-xxxxxxx"}, rp.prune, + "the leftover budget after tombstones goes to the oldest prunes; a zero leftover must yield zero "+ + "prunes, never the whole list") +} diff --git a/internal/gc/plan.go b/internal/gc/plan.go index a78791b..510fa5f 100644 --- a/internal/gc/plan.go +++ b/internal/gc/plan.go @@ -13,19 +13,31 @@ type Plan struct { Reason string } -func sortNewestFirst(ds []Deploy) { - sort.SliceStable(ds, func(i, j int) bool { - if !ds[i].Mtime.Equal(ds[j].Mtime) { - return ds[i].Mtime.After(ds[j].Mtime) - } - return ds[i].ID > ds[j].ID - }) +func capOldest[T any](items []T, n int, older func(a, b T) bool) []T { + if n <= 0 { + return nil + } + if len(items) <= n { + return items + } + out := make([]T, len(items)) + copy(out, items) + sort.SliceStable(out, func(i, j int) bool { return older(out[i], out[j]) }) + return out[:n] +} + +func olderDeploy(a, b Deploy) bool { + if !a.Mtime.Equal(b.Mtime) { + return a.Mtime.Before(b.Mtime) + } + return a.ID < b.ID } +func olderID(a, b string) bool { return a < b } + func PlanSite(site string, in RetainInput, p Policy, blastCap int) Plan { _, del := Retain(in, p) del = append(del, in.Expired...) - sortNewestFirst(del) plan := Plan{Site: site} if len(del) > 0 && blastCap <= 0 { @@ -37,7 +49,7 @@ func PlanSite(site string, in RetainInput, p Policy, blastCap int) Plan { if blastCap > 0 && len(del) > blastCap { plan.Aborted = true plan.Reason = fmt.Sprintf("delete plan of %d exceeds blast-cap %d; reaping oldest %d this run", len(del), blastCap, blastCap) - del = del[len(del)-blastCap:] + del = capOldest(del, blastCap, olderDeploy) } plan.Delete = del for _, d := range del { diff --git a/internal/gc/plan_test.go b/internal/gc/plan_test.go index 30dd9a4..8e7a8d9 100644 --- a/internal/gc/plan_test.go +++ b/internal/gc/plan_test.go @@ -92,7 +92,7 @@ func TestPlanSite_CapReapsTheOldestAcrossBothDeleteSources(t *testing.T) { plan := PlanSite("www", in, testPolicy(), 2) require.True(t, plan.Aborted) - assert.Equal(t, []string{"pend-oldest", "d-old"}, planIDs(plan), + assert.Equal(t, []string{"d-old", "pend-oldest"}, planIDs(plan), "Retain returns newest-first while ExpiredPendingDeploys returns oldest-first, and the cap slices "+ "the tail; concatenating them unsorted makes the tail the NEWEST abandoned sessions while the "+ "reason string still claims it reaped the oldest, starving retention on any site over the cap") diff --git a/internal/gc/reconcile.go b/internal/gc/reconcile.go index 82658e8..4a1c4a5 100644 --- a/internal/gc/reconcile.go +++ b/internal/gc/reconcile.go @@ -229,12 +229,8 @@ func (rc *Reconciler) applyBlastCap(ctx context.Context, site string, plan *repa report.Capped = true report.CapReason = fmt.Sprintf("destructive plan of %d exceeds blast-cap %d; attempting %d this run", destructive, rc.BlastCap, rc.BlastCap) - if len(plan.tombstone) >= rc.BlastCap { - plan.tombstone = plan.tombstone[:rc.BlastCap] - plan.prune = nil - } else { - plan.prune = plan.prune[:rc.BlastCap-len(plan.tombstone)] - } + plan.tombstone = capOldest(plan.tombstone, rc.BlastCap, olderID) + plan.prune = capOldest(plan.prune, rc.BlastCap-len(plan.tombstone), olderID) slog.WarnContext(ctx, "reconcile.capped", "site", site, "reason", report.CapReason) } From 2e15f45e38cf57fb7ec39adb21a263ba64a554a4 Mon Sep 17 00:00:00 2001 From: Mrugesh Mohapatra Date: Mon, 17 Aug 2026 17:50:16 +0530 Subject: [PATCH 26/41] fix(gc): cap the purge and survive per-site failures --- cmd/artemis/gcwire.go | 1 + cmd/artemis/gcwire_test.go | 2 + internal/gc/errorpath_test.go | 1 + internal/gc/tombstone.go | 30 +++++++++- internal/gc/tombstone_cap_test.go | 97 +++++++++++++++++++++++++++++++ internal/gc/tombstone_test.go | 1 + 6 files changed, 130 insertions(+), 2 deletions(-) create mode 100644 internal/gc/tombstone_cap_test.go diff --git a/cmd/artemis/gcwire.go b/cmd/artemis/gcwire.go index 2097ff6..36cd6bb 100644 --- a/cmd/artemis/gcwire.go +++ b/cmd/artemis/gcwire.go @@ -248,6 +248,7 @@ func newGCWiring(cfg *config.Config, repo *pg.Repo, r2c *r2.Client) (*gcWiring, Deleter: r2c, Recovery: time.Duration(cfg.Cleanup.RecoveryDays) * 24 * time.Hour, TrashBase: cfg.Cleanup.TrashPrefix, + BlastCap: cfg.Cleanup.BlastCap, Now: time.Now, Locker: repo, Audit: gcPurgeAuditor{repo: repo}, diff --git a/cmd/artemis/gcwire_test.go b/cmd/artemis/gcwire_test.go index 663db99..fef557d 100644 --- a/cmd/artemis/gcwire_test.go +++ b/cmd/artemis/gcwire_test.go @@ -225,6 +225,8 @@ func TestNewGCWiring_PlumbsBlastCapAndPrefixes(t *testing.T) { assert.Equal(t, 5, w.SiteGC.BlastCap, "BlastCap=0 would disable the mass-delete safety cap") assert.Equal(t, 7*24*time.Hour, w.SiteGC.Policy.Retention, "policy retention must derive from RetentionDays") assert.Equal(t, "_trash/", w.Purge.TrashBase, "purge must scan the configured trash base") + assert.Equal(t, 5, w.Purge.BlastCap, + "the irreversible job shares the configured ceiling; an unwired cap defaults to 0 which refuses every hard delete") assert.Equal(t, 3*24*time.Hour, w.Purge.Recovery, "recovery window must derive from RecoveryDays") require.NotNil(t, w.SiteGC.DeployPrefix) diff --git a/internal/gc/errorpath_test.go b/internal/gc/errorpath_test.go index d44f21d..9a81e2e 100644 --- a/internal/gc/errorpath_test.go +++ b/internal/gc/errorpath_test.go @@ -53,6 +53,7 @@ func newErrPurge(reaper TombstoneReaper, del Deleter) *TombstonePurge { Recovery: 7 * 24 * time.Hour, TrashBase: "_trash/", Now: func() time.Time { return testNow }, + BlastCap: defaultTestBlastCap, } } diff --git a/internal/gc/tombstone.go b/internal/gc/tombstone.go index 89cb5b2..ece8993 100644 --- a/internal/gc/tombstone.go +++ b/internal/gc/tombstone.go @@ -2,6 +2,7 @@ package gc import ( "context" + "errors" "fmt" "log/slog" "time" @@ -39,6 +40,7 @@ type TombstonePurge struct { Now func() time.Time Locker SiteLocker Audit PurgeAuditor + BlastCap int } func (p *TombstonePurge) withLock(ctx context.Context, site string, fn func() error) error { @@ -72,6 +74,18 @@ func (p *TombstonePurge) Run(ctx context.Context, dryRun bool) (PurgeResult, err if err != nil { return res, fmt.Errorf("tombstone-purge: list expired: %w", err) } + if !dryRun && len(expired) > 0 { + if p.BlastCap <= 0 { + slog.WarnContext(ctx, "gc.tombstone-purge.capped", "expired", len(expired), + "reason", "refusing every hard delete: blast-cap 0 means no ceiling was configured") + expired = nil + } else if len(expired) > p.BlastCap { + slog.WarnContext(ctx, "gc.tombstone-purge.capped", "expired", len(expired), "cap", p.BlastCap, + "reason", "hard-deleting the most overdue trash first; the remainder waits for the next run") + expired = capOldest(expired, p.BlastCap, olderTombstone) + } + } + var runErrs []error for _, t := range expired { label := t.Site + "/" + t.ID if dryRun { @@ -91,7 +105,9 @@ func (p *TombstonePurge) Run(ctx context.Context, dryRun bool) (PurgeResult, err return nil }) if lockErr != nil { - return res, lockErr + slog.WarnContext(ctx, "gc.tombstone-purge.site_failed", "site", t.Site, "deploy_id", t.ID, "err", lockErr) + runErrs = append(runErrs, lockErr) + continue } if !cleared { continue @@ -106,5 +122,15 @@ func (p *TombstonePurge) Run(ctx context.Context, dryRun bool) (PurgeResult, err } slog.InfoContext(ctx, "gc.tombstone-purge.done", "purged", len(res.Purged), "bytes", res.BytesReclaimed, "dryRun", dryRun) - return res, nil + return res, errors.Join(runErrs...) +} + +func olderTombstone(a, b Tombstone) bool { + if !a.TrashedAt.Equal(b.TrashedAt) { + return a.TrashedAt.Before(b.TrashedAt) + } + if a.Site != b.Site { + return a.Site < b.Site + } + return a.ID < b.ID } diff --git a/internal/gc/tombstone_cap_test.go b/internal/gc/tombstone_cap_test.go new file mode 100644 index 0000000..9c64720 --- /dev/null +++ b/internal/gc/tombstone_cap_test.go @@ -0,0 +1,97 @@ +package gc + +import ( + "context" + "errors" + "strings" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func expiredThree() []Tombstone { + return []Tombstone{ + {Site: "www", ID: "d-mid", TrashedAt: ago(9 * 24 * time.Hour), Bytes: 10}, + {Site: "www", ID: "d-oldest", TrashedAt: ago(30 * 24 * time.Hour), Bytes: 20}, + {Site: "learn", ID: "d-newest", TrashedAt: ago(8 * 24 * time.Hour), Bytes: 30}, + } +} + +func TestTombstonePurge_CapsTheRunAtTheOldestN(t *testing.T) { + reaper := &fakeReaper{tombstones: expiredThree()} + del := &fakeDeleter{} + p := newPurge(reaper, del) + p.BlastCap = 2 + + res, err := p.Run(context.Background(), false) + require.NoError(t, err) + + assert.Equal(t, []string{"www/d-oldest", "www/d-mid"}, res.Purged, + "the only irreversible job was the only destructive path without a ceiling; over-cap runs hard-"+ + "delete the most overdue trash first and leave the rest for tomorrow") + assert.Len(t, reaper.cleared, 2) + for _, c := range reaper.cleared { + assert.NotContains(t, c, "d-newest", "the newest expired tombstone survives to the next run") + } +} + +func TestTombstonePurge_ZeroCapRefusesEveryHardDelete(t *testing.T) { + reaper := &fakeReaper{tombstones: expiredThree()} + del := &fakeDeleter{} + p := newPurge(reaper, del) + p.BlastCap = 0 + + res, err := p.Run(context.Background(), false) + require.NoError(t, err) + + assert.Empty(t, res.Purged) + assert.Empty(t, del.deleted, + "cap 0 means no ceiling was configured; the convention everywhere else is refuse, and the one "+ + "operation that cannot be undone must not be the exception") +} + +func TestTombstonePurge_DryRunReportsEverythingRegardlessOfCap(t *testing.T) { + reaper := &fakeReaper{tombstones: expiredThree()} + p := newPurge(reaper, &fakeDeleter{}) + p.BlastCap = 1 + + res, err := p.Run(context.Background(), true) + require.NoError(t, err) + + assert.Len(t, res.Purged, 3, + "a capped REPORT hides backlog; the cap bounds destruction, never visibility") +} + +type siteFailDeleter struct { + fakeDeleter + failPrefix string +} + +func (d *siteFailDeleter) DeletePrefix(ctx context.Context, prefix string) (int, error) { + if strings.HasPrefix(prefix, d.failPrefix) { + return 0, errors.New("r2 outage for this site") + } + return d.fakeDeleter.DeletePrefix(ctx, prefix) +} + +func TestTombstonePurge_OneSiteFailureDoesNotBlockTheRest(t *testing.T) { + reaper := &fakeReaper{tombstones: []Tombstone{ + {Site: "aaa-broken", ID: "d-1", TrashedAt: ago(30 * 24 * time.Hour), Bytes: 5}, + {Site: "zzz-healthy", ID: "d-2", TrashedAt: ago(20 * 24 * time.Hour), Bytes: 7}, + }} + del := &siteFailDeleter{failPrefix: "_trash/aaa-broken/"} + p := &TombstonePurge{ + Store: reaper, Deleter: del, Recovery: 7 * 24 * time.Hour, + TrashBase: "_trash/", Now: func() time.Time { return testNow }, BlastCap: 10, + } + + res, err := p.Run(context.Background(), false) + + require.Error(t, err, "the run still reports red so the cron check-in fails") + assert.Contains(t, err.Error(), "aaa-broken/d-1") + assert.Equal(t, []string{"zzz-healthy/d-2"}, res.Purged, + "one contended or failing site used to abort the whole nightly run, silently deferring every "+ + "other site's reclamation") +} diff --git a/internal/gc/tombstone_test.go b/internal/gc/tombstone_test.go index 98d62d2..2bdc1e0 100644 --- a/internal/gc/tombstone_test.go +++ b/internal/gc/tombstone_test.go @@ -78,6 +78,7 @@ func newPurge(reaper *fakeReaper, del *fakeDeleter) *TombstonePurge { Recovery: 7 * 24 * time.Hour, TrashBase: "_trash/", Now: func() time.Time { return testNow }, + BlastCap: defaultTestBlastCap, } } From 4c1cb7d0a410ca26826411e8961319b463c08da3 Mon Sep 17 00:00:00 2001 From: Mrugesh Mohapatra Date: Mon, 17 Aug 2026 17:51:24 +0530 Subject: [PATCH 27/41] fix(gc): give both gc jobs explicit run budgets --- cmd/artemis/gcworkflows.go | 13 ++++++++----- cmd/artemis/gcworkflows_test.go | 6 ++++++ 2 files changed, 14 insertions(+), 5 deletions(-) diff --git a/cmd/artemis/gcworkflows.go b/cmd/artemis/gcworkflows.go index a2708ed..0aa3cc6 100644 --- a/cmd/artemis/gcworkflows.go +++ b/cmd/artemis/gcworkflows.go @@ -68,6 +68,7 @@ const ( cronTombstonePurge = "0 3 * * *" cronDriftDetect = "0 4 * * *" driftDetectRunBudget = 30 * time.Minute + gcRunBudget = 30 * time.Minute relayInterval = 5 * time.Second ) @@ -120,9 +121,10 @@ func gcWorkflowDefs(gcw *gcWiring, dryRun bool, sweepDrift driftSweeper) []worke })), }, { - Name: worker.WorkflowGCSite, - ConcurrencyKey: worker.ConcurrencyKeySite, - EventTriggers: []string{pg.TopicSiteChanged}, + Name: worker.WorkflowGCSite, + ConcurrencyKey: worker.ConcurrencyKeySite, + EventTriggers: []string{pg.TopicSiteChanged}, + ExecutionTimeout: gcRunBudget, Handler: observeWorkflow(worker.WorkflowGCSite, func(ctx context.Context, input map[string]any) error { site, err := siteFromInput(input) if err != nil { @@ -136,8 +138,9 @@ func gcWorkflowDefs(gcw *gcWiring, dryRun bool, sweepDrift driftSweeper) []worke }), }, { - Name: worker.WorkflowTombstonePurge, - Cron: []string{cronTombstonePurge}, + Name: worker.WorkflowTombstonePurge, + Cron: []string{cronTombstonePurge}, + ExecutionTimeout: gcRunBudget, Handler: withCheckIn(worker.WorkflowTombstonePurge, cronTombstonePurge, observeWorkflow(worker.WorkflowTombstonePurge, func(ctx context.Context, _ map[string]any) error { if _, err := gcw.Purge.Run(ctx, dryRun); err != nil { observability.CaptureBackground("tombstone.purge", err) diff --git a/cmd/artemis/gcworkflows_test.go b/cmd/artemis/gcworkflows_test.go index 9e033c0..36aac5a 100644 --- a/cmd/artemis/gcworkflows_test.go +++ b/cmd/artemis/gcworkflows_test.go @@ -193,10 +193,16 @@ func TestGCWorkflowDefs(t *testing.T) { gcSite := byName[worker.WorkflowGCSite] assert.Equal(t, worker.ConcurrencyKeySite, gcSite.ConcurrencyKey, "gc-site serialized per site (V3)") assert.Equal(t, []string{pg.TopicSiteChanged}, gcSite.EventTriggers, "gc-site triggered by the outbox topic") + assert.GreaterOrEqual(t, gcSite.ExecutionTimeout, 10*time.Minute, + "probed live: gc-site's Step.timeout is EMPTY while drift-detect carries 1800s, so gc-site runs "+ + "on the engine default; a blast-cap-sized run moves bytes object-by-object and a mid-run kill "+ + "strands tombstoned deploys at their prefixes until the next site.changed event") purge := byName[worker.WorkflowTombstonePurge] assert.Empty(t, purge.ConcurrencyKey, "tombstone-purge is global") assert.NotEmpty(t, purge.Cron, "tombstone-purge is scheduled") + assert.GreaterOrEqual(t, purge.ExecutionTimeout, 10*time.Minute, + "same gap as gc-site: the only hard-deleting job had no explicit budget either") drift := byName[workflowDriftDetect] assert.NotEmpty(t, drift.Cron, "drift-detect is cron-triggered") From 5c97934f388230faa01e6f2b4abc15a3a65b0e10 Mon Sep 17 00:00:00 2001 From: Mrugesh Mohapatra Date: Mon, 17 Aug 2026 17:52:56 +0530 Subject: [PATCH 28/41] fix(backfill): render R2 keys from the configured layout --- cmd/artemis/gcwire.go | 10 +++++++++- cmd/artemis/main.go | 20 +++++++++++++++++++- internal/backfill/backfill.go | 26 +++++++++++++++++++++----- internal/backfill/backfill_test.go | 28 ++++++++++++++++++++++++++++ 4 files changed, 77 insertions(+), 7 deletions(-) diff --git a/cmd/artemis/gcwire.go b/cmd/artemis/gcwire.go index 36cd6bb..6c118a2 100644 --- a/cmd/artemis/gcwire.go +++ b/cmd/artemis/gcwire.go @@ -139,7 +139,7 @@ func siteSegment(format string) (string, error) { return format[:slash], nil } -func newLiveAliasReader(getter aliasGetter, deployFormat string, formats ...string) (func(context.Context, string) (map[string]struct{}, error), error) { +func aliasTails(deployFormat string, formats ...string) ([]string, error) { deploySeg, err := siteSegment(deployFormat) if err != nil { return nil, fmt.Errorf("DEPLOY_PREFIX_FORMAT: %w", err) @@ -169,6 +169,14 @@ func newLiveAliasReader(getter aliasGetter, deployFormat string, formats ...stri } tails = append(tails, tail) } + return tails, nil +} + +func newLiveAliasReader(getter aliasGetter, deployFormat string, formats ...string) (func(context.Context, string) (map[string]struct{}, error), error) { + tails, err := aliasTails(deployFormat, formats...) + if err != nil { + return nil, err + } return func(ctx context.Context, dirname string) (map[string]struct{}, error) { out := map[string]struct{}{} for _, tail := range tails { diff --git a/cmd/artemis/main.go b/cmd/artemis/main.go index 62b9d74..7db2529 100644 --- a/cmd/artemis/main.go +++ b/cmd/artemis/main.go @@ -265,7 +265,25 @@ func runWith(rootCtx context.Context, cfg *config.Config) error { if pgRepo == nil { return fmt.Errorf("BACKFILL_ON_BOOT set but DATABASE_URL is unset") } - res, err := (&backfill.Backfill{Lister: r2Client, Indexer: pgRepo, Now: time.Now}).Run(rootCtx) + layout, layoutErr := newGCLayout(cfg.DeployPrefixFormat, cfg.Cleanup.TrashPrefix) + if layoutErr != nil { + return fmt.Errorf("backfill layout: %w", layoutErr) + } + tails, tailErr := aliasTails(cfg.DeployPrefixFormat, + cfg.Aliases.ProductionKeyFormat, cfg.Aliases.PreviewKeyFormat) + if tailErr != nil { + return fmt.Errorf("backfill alias formats: %w", tailErr) + } + res, err := (&backfill.Backfill{ + Lister: r2Client, Indexer: pgRepo, Now: time.Now, + SitePrefix: layout.sitePrefix, + AliasKey: func(dirname, mode string) string { + if mode == "production" { + return dirname + "/" + tails[0] + } + return dirname + "/" + tails[1] + }, + }).Run(rootCtx) if err != nil { return fmt.Errorf("backfill: %w", err) } diff --git a/internal/backfill/backfill.go b/internal/backfill/backfill.go index cdcbfad..dcebec8 100644 --- a/internal/backfill/backfill.go +++ b/internal/backfill/backfill.go @@ -26,9 +26,25 @@ type Indexer interface { } type Backfill struct { - Lister Lister - Indexer Indexer - Now func() time.Time + Lister Lister + Indexer Indexer + Now func() time.Time + SitePrefix func(dirname string) string + AliasKey func(dirname, mode string) string +} + +func (b *Backfill) sitePrefix(dirname string) string { + if b.SitePrefix != nil { + return b.SitePrefix(dirname) + } + return dirname + "/deploys/" +} + +func (b *Backfill) aliasKey(dirname, mode string) string { + if b.AliasKey != nil { + return b.AliasKey(dirname, mode) + } + return dirname + "/" + mode } type Result struct { @@ -54,7 +70,7 @@ func (b *Backfill) Run(ctx context.Context) (Result, error) { for _, site := range sites { res.Sites++ - deploysPrefix := site + "/deploys/" + deploysPrefix := b.sitePrefix(site) keys, err := b.Lister.ListPrefix(ctx, deploysPrefix) if err != nil { return res, fmt.Errorf("backfill: list %s: %w", site, err) @@ -100,7 +116,7 @@ func (b *Backfill) Run(ctx context.Context) (Result, error) { } for _, mode := range []string{"production", "preview"} { - v, err := b.Lister.GetAlias(ctx, site+"/"+mode) + v, err := b.Lister.GetAlias(ctx, b.aliasKey(site, mode)) if err != nil { if r2.IsNotFound(err) { continue diff --git a/internal/backfill/backfill_test.go b/internal/backfill/backfill_test.go index 11c8a80..94955b4 100644 --- a/internal/backfill/backfill_test.go +++ b/internal/backfill/backfill_test.go @@ -310,3 +310,31 @@ func TestBackfill_AliasKeyIsR2DirRelative(t *testing.T) { assert.Equal(t, 2, res.Aliases, "alias key is the R2-dir-relative literal /; the dir from ListSites already carries the .freecode.camp suffix, so the slug-templated ALIAS_*_KEY_FORMAT must NOT be re-applied") } + +func TestBackfill_RendersPrefixesFromTheConfiguredLayout(t *testing.T) { + lister := &fakeLister{ + sites: []string{"test.freecode.camp"}, + byPfx: map[string][]string{ + "test.freecode.camp/builds/": { + "test.freecode.camp/builds/20260101-000000-abc1234/index.html", + }, + }, + bytesByPfx: map[string]int64{"test.freecode.camp/builds/20260101-000000-abc1234/": 42}, + aliases: map[string]string{"test.freecode.camp/production": "20260101-000000-abc1234"}, + } + idx := &fakeIndexer{} + b := &Backfill{ + Lister: lister, Indexer: idx, Now: func() time.Time { return time.Unix(0, 0) }, + SitePrefix: func(dirname string) string { return dirname + "/builds/" }, + AliasKey: func(dirname, mode string) string { return dirname + "/" + mode }, + } + + res, err := b.Run(context.Background()) + require.NoError(t, err) + + assert.Equal(t, 1, res.Deploys, + "the old code hardcoded /deploys/ and /, ignoring DEPLOY_PREFIX_FORMAT and the "+ + "alias key formats entirely; under any layout whose sub-path differs it indexed zero deploys "+ + "and reported success") + assert.Equal(t, 1, res.Aliases) +} From 778f00bea837a38e9f7e39d014dfd1967e522cb5 Mon Sep 17 00:00:00 2001 From: Mrugesh Mohapatra Date: Mon, 17 Aug 2026 17:57:54 +0530 Subject: [PATCH 29/41] refactor(pg): one implementation for the site lock --- internal/pg/lock.go | 19 +++---------------- 1 file changed, 3 insertions(+), 16 deletions(-) diff --git a/internal/pg/lock.go b/internal/pg/lock.go index 8b260af..8e126ad 100644 --- a/internal/pg/lock.go +++ b/internal/pg/lock.go @@ -13,25 +13,12 @@ import ( ) func (r *Repo) WithSiteLock(ctx context.Context, site string, fn func() error) error { - conn, err := pgx.ConnectConfig(ctx, r.pool.Config().ConnConfig.Copy()) + sess, err := r.NewLockSession(ctx) if err != nil { - return fmt.Errorf("site lock %s: connect: %w", site, err) - } - defer func() { - closeCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 5*time.Second) - defer cancel() - if err := conn.Close(closeCtx); err != nil { - slog.WarnContext(ctx, "lock.site.close_failed", "site", site, "err", err) - } - }() - - if _, err := conn.Exec(ctx, `SET lock_timeout = '30s'`); err != nil { - return fmt.Errorf("site lock %s: set lock_timeout: %w", site, err) - } - if _, err := conn.Exec(ctx, `SELECT pg_advisory_lock(hashtextextended($1, 0))`, site); err != nil { return fmt.Errorf("site lock %s: %w", site, err) } - return fn() + defer sess.Close(ctx) + return sess.WithSiteLock(ctx, site, fn) } func (r *Repo) NewLockSession(ctx context.Context) (gc.LockSession, error) { From 5c1cc74d620517dd3c9b2cb743abdd212e76890a Mon Sep 17 00:00:00 2001 From: Mrugesh Mohapatra Date: Mon, 17 Aug 2026 17:58:54 +0530 Subject: [PATCH 30/41] fix(auth): bound the in-process identity caches --- internal/auth/github.go | 37 ++++++++++++++++---- internal/auth/github_cachebound_test.go | 46 +++++++++++++++++++++++++ 2 files changed, 77 insertions(+), 6 deletions(-) create mode 100644 internal/auth/github_cachebound_test.go diff --git a/internal/auth/github.go b/internal/auth/github.go index a0ab883..7a5ec0d 100644 --- a/internal/auth/github.go +++ b/internal/auth/github.go @@ -200,6 +200,7 @@ func (c *GitHubClient) fetchUser(ctx context.Context, cacheKey, token string) (s } c.mu.Lock() + pruneMap(c.userCache, func(e userCacheEntry) bool { return c.expiredEntry(e.expires) }, maxCacheEntries) c.userCache[cacheKey] = userCacheEntry{ login: u.Login, expires: c.now().Add(c.cfg.CacheTTL), @@ -222,12 +223,39 @@ func hashToken(token string) string { // configured TTL and 30s. const negCacheCap = 30 * time.Second +// maxCacheEntries bounds each in-process cache map. Entries were only +// ever logically expired, never deleted, so every distinct token and +// (user, team) pair accumulated for the life of the process. +const maxCacheEntries = 4096 + +func pruneMap[K comparable, V any](m map[K]V, expired func(V) bool, limit int) { + if len(m) < limit { + return + } + for k, v := range m { + if expired(v) { + delete(m, k) + } + } + for k := range m { + if len(m) < limit { + break + } + delete(m, k) + } +} + +func (c *GitHubClient) expiredEntry(expires time.Time) bool { + return !expires.After(c.now()) +} + func (c *GitHubClient) cacheNegative(key string, err error) { ttl := c.cfg.CacheTTL if ttl > negCacheCap { ttl = negCacheCap } c.mu.Lock() + pruneMap(c.userCache, func(e userCacheEntry) bool { return c.expiredEntry(e.expires) }, maxCacheEntries) c.userCache[key] = userCacheEntry{ err: err, expires: c.now().Add(ttl), @@ -316,6 +344,7 @@ func (c *GitHubClient) fetchTeamMembership(ctx context.Context, token, user, tea } c.mu.Lock() + pruneMap(c.teamCache, func(e teamCacheEntry) bool { return c.expiredEntry(e.expires) }, maxCacheEntries) c.teamCache[key] = teamCacheEntry{ member: member, expires: c.now().Add(c.cfg.CacheTTL), @@ -383,6 +412,7 @@ func (c *GitHubClient) userTeamsThroughDurableCache(ctx context.Context, cacheKe func (c *GitHubClient) storeUserTeams(cacheKey string, teams []string) { c.mu.Lock() + pruneMap(c.userTeamsCache, func(e userTeamsCacheEntry) bool { return c.expiredEntry(e.expires) }, maxCacheEntries) c.userTeamsCache[cacheKey] = userTeamsCacheEntry{ teams: append([]string(nil), teams...), expires: c.now().Add(c.cfg.CacheTTL), @@ -453,12 +483,7 @@ func (c *GitHubClient) fetchUserTeams(ctx context.Context, cacheKey, token strin page++ } - c.mu.Lock() - c.userTeamsCache[cacheKey] = userTeamsCacheEntry{ - teams: append([]string(nil), teams...), - expires: c.now().Add(c.cfg.CacheTTL), - } - c.mu.Unlock() + c.storeUserTeams(cacheKey, teams) return teams, nil } diff --git a/internal/auth/github_cachebound_test.go b/internal/auth/github_cachebound_test.go new file mode 100644 index 0000000..e04e52d --- /dev/null +++ b/internal/auth/github_cachebound_test.go @@ -0,0 +1,46 @@ +package auth + +import ( + "fmt" + "testing" + "time" + + "github.com/stretchr/testify/assert" +) + +func TestGitHubClient_CachesAreBoundedAcrossTokenChurn(t *testing.T) { + clock := time.Unix(0, 0) + c := NewGitHubClient(GitHubClientConfig{Now: func() time.Time { return clock }}) + + for i := 0; i < maxCacheEntries*2; i++ { + c.cacheNegative(fmt.Sprintf("hash-%d", i), ErrGitHubUnauthenticated) + } + + assert.LessOrEqual(t, len(c.userCache), maxCacheEntries, + "every distinct bearer ever presented used to leave a permanent map entry; CI tokens rotate and "+ + "pods live for weeks, so the maps grew monotonically for the life of the process") + + c.mu.Lock() + for i := 0; i < maxCacheEntries*2; i++ { + key := teamCacheKey{user: fmt.Sprintf("u-%d", i), team: "t"} + pruneMap(c.teamCache, func(e teamCacheEntry) bool { return !e.expires.After(c.now()) }, maxCacheEntries) + c.teamCache[key] = teamCacheEntry{member: true, expires: clock.Add(time.Minute)} + } + c.mu.Unlock() + assert.LessOrEqual(t, len(c.teamCache), maxCacheEntries+1) +} + +func TestPruneMap_DropsExpiredBeforeLive(t *testing.T) { + clock := time.Unix(1000, 0) + m := map[string]userCacheEntry{ + "dead-1": {expires: clock.Add(-time.Second)}, + "dead-2": {expires: clock.Add(-time.Minute)}, + "live-1": {expires: clock.Add(time.Hour)}, + } + + pruneMap(m, func(e userCacheEntry) bool { return !e.expires.After(clock) }, 2) + + assert.Len(t, m, 1) + _, ok := m["live-1"] + assert.True(t, ok, "expired entries go first; a live entry is evicted only when the cap still overflows") +} From 7036f494be5c4c7008cd44e07d2dde03ab45e81a Mon Sep 17 00:00:00 2001 From: Mrugesh Mohapatra Date: Mon, 17 Aug 2026 18:00:13 +0530 Subject: [PATCH 31/41] fix(server): give upload and finalize a deadline --- internal/server/server.go | 11 +++++ internal/server/timeout_test.go | 79 +++++++++++++++++++++++++++++++++ 2 files changed, 90 insertions(+) create mode 100644 internal/server/timeout_test.go diff --git a/internal/server/server.go b/internal/server/server.go index 8542491..8e03870 100644 --- a/internal/server/server.go +++ b/internal/server/server.go @@ -41,10 +41,20 @@ import ( const apiRequestTimeout = 60 * time.Second +// uploadRequestTimeout bounds the JWT-guarded upload/finalize routes, +// which previously ran with no deadline at all. Sized for the 100 MiB +// default body cap over a slow residential uplink, not for the 60s +// interactive budget the bearer routes get. +const uploadRequestTimeout = 10 * time.Minute + // New returns a chi router fully wired with the Handlers' endpoints + // the standard middleware chain (Sentry → RequestID → AccessLog → // Recoverer). func New(h *handler.Handlers) http.Handler { + return newWithUploadTimeout(h, uploadRequestTimeout) +} + +func newWithUploadTimeout(h *handler.Handlers, uploadTimeout time.Duration) http.Handler { r := chi.NewRouter() // Mount the Sentry request middleware only when a client is actually // configured (Init ran with a DSN). When Sentry is disabled this adds @@ -101,6 +111,7 @@ func New(h *handler.Handlers) http.Handler { // Deploy-session JWT branch — scoped to (login, site, deployId). r.Group(func(r chi.Router) { r.Use(h.RequireDeployJWT) + r.Use(middleware.Timeout(uploadTimeout)) r.Put("/deploy/{deployId}/upload", h.DeployUpload) r.Post("/deploy/{deployId}/finalize", h.DeployFinalize) }) diff --git a/internal/server/timeout_test.go b/internal/server/timeout_test.go new file mode 100644 index 0000000..28e7979 --- /dev/null +++ b/internal/server/timeout_test.go @@ -0,0 +1,79 @@ +package server + +import ( + "context" + "io" + "net/http" + "net/http/httptest" + "testing" + "time" + + "github.com/freeCodeCamp/artemis/internal/auth" + "github.com/freeCodeCamp/artemis/internal/handler" + "github.com/freeCodeCamp/artemis/internal/registry" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +type stubSnapshot struct{} + +func (stubSnapshot) Sites() []string { return []string{"www"} } +func (stubSnapshot) TeamsForSite(string) []string { return []string{"team-eng"} } + +type stubSites struct{} + +func (stubSites) Snapshot() registry.Snapshot { return stubSnapshot{} } + +type blockingR2 struct{} + +func (blockingR2) PutObject(ctx context.Context, _ string, _ io.Reader, _ string, _ int64) error { + <-ctx.Done() + return ctx.Err() +} +func (blockingR2) PutAlias(context.Context, string, string) error { return nil } +func (blockingR2) GetAlias(context.Context, string) (string, error) { return "", nil } +func (blockingR2) ListPrefix(context.Context, string) ([]string, error) { return nil, nil } +func (blockingR2) HasPrefix(context.Context, string) (bool, error) { return false, nil } +func (blockingR2) HasObject(context.Context, string) (bool, error) { return false, nil } +func (blockingR2) VerifyDeployComplete(context.Context, string, []string) error { + return nil +} +func (blockingR2) MovePrefix(context.Context, string, string) (int, error) { return 0, nil } +func (blockingR2) PrefixBytes(context.Context, string) (int64, error) { return 0, nil } + +func TestRouter_UploadCarriesADeadline(t *testing.T) { + signer, err := auth.NewDeploySessionSigner("0123456789abcdef0123456789abcdef", 15*time.Minute) + require.NoError(t, err) + tok, _, err := signer.Sign("alice", "www", "20260101-000000-abc1234") + require.NoError(t, err) + + h := &handler.Handlers{ + JWT: signer, + Sites: stubSites{}, + R2: blockingR2{}, + DeployPrefix: mustTemplate(t), + Now: time.Now, + } + r := newWithUploadTimeout(h, 50*time.Millisecond) + + req := httptest.NewRequest(http.MethodPut, "/api/deploy/20260101-000000-abc1234/upload?path=index.html", nil) + req.Header.Set("Authorization", "Bearer "+tok) + w := httptest.NewRecorder() + + done := make(chan struct{}) + go func() { r.ServeHTTP(w, req); close(done) }() + select { + case <-done: + case <-time.After(5 * time.Second): + t.Fatal("upload hung: the deploy-session JWT group had no request timeout at all, so a stalled " + + "R2 put held the goroutine and connection for as long as the client stayed connected") + } + assert.NotEqual(t, http.StatusOK, w.Code) +} + +func mustTemplate(t *testing.T) handler.DeployPrefixTemplate { + t.Helper() + tpl, err := handler.NewDeployPrefixTemplate("/deploys/-/") + require.NoError(t, err) + return tpl +} From 82fe4083d64975a64cfc3f3ee71f74f307d6740b Mon Sep 17 00:00:00 2001 From: Mrugesh Mohapatra Date: Mon, 17 Aug 2026 18:07:26 +0530 Subject: [PATCH 32/41] chore: delete the dead background-plane code --- cmd/artemis/gcwire.go | 2 - cmd/artemis/gcworkflows.go | 5 +- cmd/artemis/gcworkflows_test.go | 10 +- cmd/artemis/wire_pgrepo_test.go | 2 +- cmd/artemis/workflowerrors_test.go | 4 +- internal/handler/deploy.go | 2 - internal/handler/handler.go | 23 ----- internal/handler/outbox_emit_test.go | 89 ----------------- internal/handler/pg_writethrough_test.go | 6 -- internal/handler/site.go | 4 - internal/handler/sitekey_dirname_test.go | 12 --- internal/pg/alias.go | 39 -------- internal/pg/alias_create_test.go | 49 ---------- internal/pg/alias_test.go | 64 ------------ internal/pg/migrate_test.go | 7 +- internal/pg/outbox.go | 22 ----- internal/pg/outbox_markpublished_test.go | 9 +- internal/pg/outbox_relaybatch_test.go | 9 +- internal/pg/outbox_test.go | 32 ++++-- internal/pg/saga_test.go | 3 +- internal/server/server.go | 1 + internal/teamcache/teamcache.go | 16 --- internal/teamcache/teamcache_errors_test.go | 25 ----- internal/teamcache/teamcache_test.go | 74 -------------- internal/worker/debounce.go | 62 ------------ internal/worker/debounce_test.go | 103 -------------------- internal/worker/deployflows.go | 21 ---- internal/worker/deployflows_test.go | 32 ------ internal/worker/runtime.go | 4 +- internal/worker/runtime_test.go | 2 +- 30 files changed, 47 insertions(+), 686 deletions(-) delete mode 100644 internal/handler/outbox_emit_test.go delete mode 100644 internal/pg/alias.go delete mode 100644 internal/pg/alias_create_test.go delete mode 100644 internal/pg/alias_test.go delete mode 100644 internal/teamcache/teamcache_errors_test.go delete mode 100644 internal/worker/debounce.go delete mode 100644 internal/worker/debounce_test.go delete mode 100644 internal/worker/deployflows.go delete mode 100644 internal/worker/deployflows_test.go diff --git a/cmd/artemis/gcwire.go b/cmd/artemis/gcwire.go index 6c118a2..6ace1eb 100644 --- a/cmd/artemis/gcwire.go +++ b/cmd/artemis/gcwire.go @@ -23,7 +23,6 @@ type auditRecorder interface { var captureAuditFailure = observability.CaptureBackground var ( - _ handler.SiteChangeEmitter = (*pg.Repo)(nil) _ handler.TombstoneStore = (*pg.Repo)(nil) _ handler.TrashStore = (*pg.Repo)(nil) _ handler.DeployIndexWriter = (*pg.Repo)(nil) @@ -39,7 +38,6 @@ func wirePGRepo(h *handler.Handlers, repo *pg.Repo) { if repo == nil { return } - h.Outbox = repo h.Tombstones = repo h.Trash = repo h.Index = repo diff --git a/cmd/artemis/gcworkflows.go b/cmd/artemis/gcworkflows.go index 0aa3cc6..2ffb3e6 100644 --- a/cmd/artemis/gcworkflows.go +++ b/cmd/artemis/gcworkflows.go @@ -64,7 +64,6 @@ func newRunID() string { } const ( - workflowDriftDetect = "drift-detect" cronTombstonePurge = "0 3 * * *" cronDriftDetect = "0 4 * * *" driftDetectRunBudget = 30 * time.Minute @@ -108,10 +107,10 @@ func observeWorkflow(name string, fn worker.Handler) worker.Handler { func gcWorkflowDefs(gcw *gcWiring, dryRun bool, sweepDrift driftSweeper) []worker.WorkflowDef { return []worker.WorkflowDef{ { - Name: workflowDriftDetect, + Name: worker.WorkflowDriftDetect, Cron: []string{cronDriftDetect}, ExecutionTimeout: driftDetectRunBudget, - Handler: withCheckIn(workflowDriftDetect, cronDriftDetect, observeWorkflow(workflowDriftDetect, func(ctx context.Context, _ map[string]any) error { + Handler: withCheckIn(worker.WorkflowDriftDetect, cronDriftDetect, observeWorkflow(worker.WorkflowDriftDetect, func(ctx context.Context, _ map[string]any) error { res, err := sweepDrift(ctx) if err != nil { captureBackground(opDriftSweep, err) diff --git a/cmd/artemis/gcworkflows_test.go b/cmd/artemis/gcworkflows_test.go index 36aac5a..b4fd8dc 100644 --- a/cmd/artemis/gcworkflows_test.go +++ b/cmd/artemis/gcworkflows_test.go @@ -53,12 +53,12 @@ func TestCronCheckIn_DriftDetectAndPurge(t *testing.T) { byName[d.Name] = d } - require.NoError(t, byName[workflowDriftDetect].Handler(context.Background(), nil)) + require.NoError(t, byName[worker.WorkflowDriftDetect].Handler(context.Background(), nil)) require.NoError(t, byName[worker.WorkflowTombstonePurge].Handler(context.Background(), nil)) require.Len(t, got, 4, "two check-ins (in_progress+ok) per cron workflow") - assert.Equal(t, ci{workflowDriftDetect, cronDriftDetect, sentry.CheckInStatusInProgress}, got[0]) - assert.Equal(t, workflowDriftDetect, got[1].slug) + assert.Equal(t, ci{worker.WorkflowDriftDetect, cronDriftDetect, sentry.CheckInStatusInProgress}, got[0]) + assert.Equal(t, worker.WorkflowDriftDetect, got[1].slug) assert.Equal(t, sentry.CheckInStatusOK, got[1].status) assert.Equal(t, ci{worker.WorkflowTombstonePurge, cronTombstonePurge, sentry.CheckInStatusInProgress}, got[2]) assert.Equal(t, worker.WorkflowTombstonePurge, got[3].slug) @@ -204,7 +204,7 @@ func TestGCWorkflowDefs(t *testing.T) { assert.GreaterOrEqual(t, purge.ExecutionTimeout, 10*time.Minute, "same gap as gc-site: the only hard-deleting job had no explicit budget either") - drift := byName[workflowDriftDetect] + drift := byName[worker.WorkflowDriftDetect] assert.NotEmpty(t, drift.Cron, "drift-detect is cron-triggered") assert.GreaterOrEqual(t, drift.ExecutionTimeout, 10*time.Minute, "the sweep lists every object of every site — 22745 objects across 76 sites in production, "+ @@ -219,7 +219,7 @@ func TestGCWorkflowDefs_NoWorkflowCanRepairOnASchedule(t *testing.T) { gcw := &gcWiring{SiteGC: &gc.SiteGC{}, Purge: &gc.TombstonePurge{}, Reconciler: &gc.Reconciler{}} for _, d := range gcWorkflowDefs(gcw, true, cleanSweep) { - assert.NotEqual(t, worker.WorkflowReconcile, d.Name, + assert.NotEqual(t, "reconcile", d.Name, "reconcile repairs bytes and is human-invoked only; a schedule must never reach it") } } diff --git a/cmd/artemis/wire_pgrepo_test.go b/cmd/artemis/wire_pgrepo_test.go index a7a4443..48e03d4 100644 --- a/cmd/artemis/wire_pgrepo_test.go +++ b/cmd/artemis/wire_pgrepo_test.go @@ -15,7 +15,7 @@ func TestWirePGRepo_WiresAuditAndAllPGDeps(t *testing.T) { wirePGRepo(h, &pg.Repo{}) require.NotNil(t, h.Audit, "Audit MUST be wired or audit_log silently never persists for HTTP actions") - require.NotNil(t, h.Outbox) + require.NotNil(t, h.Pending) require.NotNil(t, h.Tombstones) require.NotNil(t, h.Trash) require.NotNil(t, h.Index) diff --git a/cmd/artemis/workflowerrors_test.go b/cmd/artemis/workflowerrors_test.go index 59359d6..e4b9317 100644 --- a/cmd/artemis/workflowerrors_test.go +++ b/cmd/artemis/workflowerrors_test.go @@ -39,7 +39,7 @@ func TestDriftDetectWorkflow_PropagatesASweepFailure(t *testing.T) { gcw := &gcWiring{SiteGC: &gc.SiteGC{}, Purge: &gc.TombstonePurge{}, Reconciler: &gc.Reconciler{}} failing := func(context.Context) (sweepResult, error) { return sweepResult{}, errors.New("list r2: down") } - def := defByName(t, gcWorkflowDefs(gcw, true, failing), workflowDriftDetect) + def := defByName(t, gcWorkflowDefs(gcw, true, failing), worker.WorkflowDriftDetect) err := def.Handler(context.Background(), nil) @@ -50,7 +50,7 @@ func TestDriftDetectWorkflow_PropagatesASweepFailure(t *testing.T) { func TestDriftDetectWorkflow_NeedsNoInput(t *testing.T) { gcw := &gcWiring{SiteGC: &gc.SiteGC{}, Purge: &gc.TombstonePurge{}, Reconciler: &gc.Reconciler{}} - def := defByName(t, gcWorkflowDefs(gcw, true, cleanSweep), workflowDriftDetect) + def := defByName(t, gcWorkflowDefs(gcw, true, cleanSweep), worker.WorkflowDriftDetect) require.NoError(t, def.Handler(context.Background(), map[string]any{}), "the sweep enumerates the fleet itself, so no producer has to name a site for it") diff --git a/internal/handler/deploy.go b/internal/handler/deploy.go index 427d901..673dba7 100644 --- a/internal/handler/deploy.go +++ b/internal/handler/deploy.go @@ -288,8 +288,6 @@ func (h *Handlers) DeployFinalize(w http.ResponseWriter, r *http.Request) { writeUpstreamError(w, r, http.StatusBadGateway, "pg_write_failed", "pg.finalize.index", err) return errAliasWriteHandled } - } else { - h.emitSiteChanged(commitCtx, claims.Site) } return nil }) diff --git a/internal/handler/handler.go b/internal/handler/handler.go index 8417cab..76d2a3f 100644 --- a/internal/handler/handler.go +++ b/internal/handler/handler.go @@ -81,10 +81,6 @@ type TrashStore interface { RestoreDeploy(ctx context.Context, site, id string, mtime time.Time, bytes int64) error } -type SiteChangeEmitter interface { - EnqueueSiteChanged(ctx context.Context, site string) error -} - type DeployIndexWriter interface { FinalizeAtomic(ctx context.Context, site, deployID, mode string, mtime time.Time, bytes int64) error AliasAtomic(ctx context.Context, site, name, deployID string, at time.Time) error @@ -127,7 +123,6 @@ type Handlers struct { TrashPrefixBase string // e.g. "_trash/" Trash TrashStore TrashRecovery time.Duration - Outbox SiteChangeEmitter Index DeployIndexWriter Pending PendingDeployWriter Locker SiteLocker @@ -171,24 +166,6 @@ func (h *Handlers) withSiteLock(ctx context.Context, dirname string, fn func() e return h.Locker.WithSiteLock(ctx, dirname, fn) } -func (h *Handlers) emitSiteChanged(ctx context.Context, site string) { - if h.Outbox == nil { - return - } - site = h.DeployPrefix.SiteDirname(site) - ctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 5*time.Second) - defer cancel() - if err := h.Outbox.EnqueueSiteChanged(ctx, site); err != nil { - slog.ErrorContext(ctx, "outbox.enqueue.failed", "site", site, "err", err) - sentry.WithScope(func(scope *sentry.Scope) { - scope.SetTag("op", "outbox.enqueue") - scope.SetTag("site", site) - scope.SetFingerprint([]string{"outbox.enqueue"}) - sentry.CaptureException(err) - }) - } -} - func (h *Handlers) auditFromScope(ctx context.Context, action, outcome string, detail map[string]any) { sc := telemetry.FromContext(ctx) h.audit(ctx, pg.AuditEvent{ diff --git a/internal/handler/outbox_emit_test.go b/internal/handler/outbox_emit_test.go deleted file mode 100644 index c43dd3f..0000000 --- a/internal/handler/outbox_emit_test.go +++ /dev/null @@ -1,89 +0,0 @@ -package handler - -import ( - "context" - "encoding/json" - "net/http" - "testing" - - "github.com/stretchr/testify/assert" - "github.com/stretchr/testify/require" -) - -type fakeOutbox struct { - sites []string -} - -func (f *fakeOutbox) EnqueueSiteChanged(_ context.Context, site string) error { - f.sites = append(f.sites, site) - return nil -} - -type ctxCapturingOutbox struct { - capturedDone bool - called bool -} - -func (f *ctxCapturingOutbox) EnqueueSiteChanged(ctx context.Context, _ string) error { - f.called = true - select { - case <-ctx.Done(): - f.capturedDone = true - default: - } - return nil -} - -func TestEmitSiteChanged_DetachedFromRequestCancellation(t *testing.T) { - h, _ := newTestHandlers(t, &fakeGH{}, standardSites(), newFakeR2()) - ob := &ctxCapturingOutbox{} - h.Outbox = ob - - ctx, cancel := context.WithCancel(context.Background()) - cancel() - - h.emitSiteChanged(ctx, "www") - - require.True(t, ob.called, "emitSiteChanged must enqueue even when the request context is canceled") - assert.False(t, ob.capturedDone, "enqueue context must be detached from the canceled request context") -} - -func TestFinalize_EmitsSiteChanged(t *testing.T) { - store := newFakeR2() - h, jwt := newTestHandlers(t, &fakeGH{}, standardSites(), store) - ob := &fakeOutbox{} - h.Outbox = ob - - deployID := "20260420-141522-abc1234" - store.objects["www/deploys/"+deployID+"/index.html"] = []byte("hi") - tok, _, err := jwt.Sign("alice", "www", deployID) - require.NoError(t, err) - - body, _ := json.Marshal(DeployFinalizeRequest{Mode: "preview", Files: []string{"index.html"}}) - w := withChiRoute(http.MethodPost, "/api/deploy/{deployId}/finalize", - "/api/deploy/"+deployID+"/finalize", - body, - map[string]string{"Authorization": "Bearer " + tok}, - h.RequireDeployJWT(http.HandlerFunc(h.DeployFinalize)).ServeHTTP, - context.Background(), - ) - require.Equal(t, http.StatusOK, w.Code, w.Body.String()) - assert.Equal(t, []string{"www"}, ob.sites, "finalize emits site.changed for event-driven GC") -} - -func TestPromote_EmitsSiteChanged(t *testing.T) { - store := newFakeR2() - store.aliases["www/preview"] = "20260420-141522-abc1234" - store.objects["www/deploys/20260420-141522-abc1234/index.html"] = []byte("hi") - h, _ := newTestHandlers(t, authedGH(), standardSites(), store) - ob := &fakeOutbox{} - h.Outbox = ob - - w := withSiteRoute(http.MethodPost, "/api/site/{site}/promote", - "/api/site/www/promote", nil, - contextWithLogin(context.Background(), "alice", "tok"), - h.SitePromote, - ) - require.Equal(t, http.StatusOK, w.Code, w.Body.String()) - assert.Equal(t, []string{"www"}, ob.sites) -} diff --git a/internal/handler/pg_writethrough_test.go b/internal/handler/pg_writethrough_test.go index 1704565..a3dbb45 100644 --- a/internal/handler/pg_writethrough_test.go +++ b/internal/handler/pg_writethrough_test.go @@ -43,8 +43,6 @@ func TestDeployFinalize_PGWriteThrough(t *testing.T) { h.DeployPrefix = mustDeployPrefixTemplate(prodShapedFormat) idx := &fakeIndex{} h.Index = idx - ob := &fakeOutbox{} - h.Outbox = ob deployID := "20260420-141522-abc1234" store.objects["www.freecode.camp/deploys/"+deployID+"/index.html"] = []byte("hi") @@ -63,7 +61,6 @@ func TestDeployFinalize_PGWriteThrough(t *testing.T) { assert.Equal(t, []string{"www.freecode.camp/" + deployID + "/preview"}, idx.finalized, "finalize must index deploy+alias+event transactionally under the dirname key") - assert.Empty(t, ob.sites, "tx path owns the outbox event; no duplicate direct emit") var wantBytes int64 for k, v := range store.objects { @@ -129,8 +126,6 @@ func TestSitePromote_PGWriteThrough(t *testing.T) { h.DeployPrefix = mustDeployPrefixTemplate(prodShapedFormat) idx := &fakeIndex{} h.Index = idx - ob := &fakeOutbox{} - h.Outbox = ob deployID := "20260420-141522-abc1234" store.objects["www.freecode.camp/deploys/"+deployID+"/index.html"] = []byte("hi") @@ -145,7 +140,6 @@ func TestSitePromote_PGWriteThrough(t *testing.T) { assert.Equal(t, []string{"www.freecode.camp/production/" + deployID}, idx.aliased, "promote must upsert the PG alias row so the GC planner sees the new pin") - assert.Empty(t, ob.sites) } func TestSiteRollback_PGWriteThrough(t *testing.T) { diff --git a/internal/handler/site.go b/internal/handler/site.go index 74737a3..0671a7f 100644 --- a/internal/handler/site.go +++ b/internal/handler/site.go @@ -155,8 +155,6 @@ func (h *Handlers) SitePromote(w http.ResponseWriter, r *http.Request) { writeUpstreamError(w, r, http.StatusBadGateway, "pg_write_failed", "pg.alias.promote", err) return errAliasWriteHandled } - } else { - h.emitSiteChanged(commitCtx, site) } return nil }) @@ -274,8 +272,6 @@ func (h *Handlers) SiteRollback(w http.ResponseWriter, r *http.Request) { writeUpstreamError(w, r, http.StatusBadGateway, "pg_write_failed", "pg.alias.rollback", err) return errAliasWriteHandled } - } else { - h.emitSiteChanged(commitCtx, site) } return nil }) diff --git a/internal/handler/sitekey_dirname_test.go b/internal/handler/sitekey_dirname_test.go index 5047edf..1f604c7 100644 --- a/internal/handler/sitekey_dirname_test.go +++ b/internal/handler/sitekey_dirname_test.go @@ -12,18 +12,6 @@ import ( const prodShapedFormat = ".freecode.camp/deploys/-/" -func TestEmitSiteChanged_CanonicalDirname(t *testing.T) { - h, _ := newTestHandlers(t, staffCallerGH(), standardSites(), newFakeR2()) - h.DeployPrefix = mustDeployPrefixTemplate(prodShapedFormat) - ob := &fakeOutbox{} - h.Outbox = ob - - h.emitSiteChanged(context.Background(), "www") - - assert.Equal(t, []string{"www.freecode.camp"}, ob.sites, - "site.changed payload must carry the R2 dirname (GC index key), not the registry slug") -} - func TestSitePurge_DirnameKeyedBytesAndTombstone(t *testing.T) { store := newFakeR2() store.objects["example.freecode.camp/deploys/20260420-141522-abc1234/index.html"] = []byte("hi") diff --git a/internal/pg/alias.go b/internal/pg/alias.go deleted file mode 100644 index 9f63683..0000000 --- a/internal/pg/alias.go +++ /dev/null @@ -1,39 +0,0 @@ -package pg - -import ( - "context" - "errors" - "fmt" - "time" - - "github.com/jackc/pgx/v5" -) - -func (r *Repo) SetAliasCAS(ctx context.Context, site, name, expected, next string, at time.Time) (current string, ok bool, err error) { - err = r.WithTx(ctx, func(tx pgx.Tx) error { - var cur string - scanErr := tx.QueryRow(ctx, - `SELECT deploy_id FROM aliases WHERE site = $1 AND name = $2 FOR UPDATE`, site, name).Scan(&cur) - if scanErr != nil && !errors.Is(scanErr, pgx.ErrNoRows) { - return fmt.Errorf("alias cas read %s/%s: %w", site, name, scanErr) - } - current = cur - if cur != expected { - ok = false - return nil - } - if _, err := tx.Exec(ctx, - `INSERT INTO aliases (site, name, deploy_id, updated_at) - VALUES ($1, $2, $3, $4) - ON CONFLICT (site, name) DO UPDATE SET deploy_id = EXCLUDED.deploy_id, updated_at = EXCLUDED.updated_at`, - site, name, next, at); err != nil { - return fmt.Errorf("alias cas write %s/%s: %w", site, name, err) - } - if err := Enqueue(ctx, tx, TopicSiteChanged, map[string]string{"site": site}); err != nil { - return err - } - ok = true - return nil - }) - return current, ok, err -} diff --git a/internal/pg/alias_create_test.go b/internal/pg/alias_create_test.go deleted file mode 100644 index 999b7fb..0000000 --- a/internal/pg/alias_create_test.go +++ /dev/null @@ -1,49 +0,0 @@ -package pg - -import ( - "context" - "testing" - "time" - - "github.com/stretchr/testify/assert" - "github.com/stretchr/testify/require" -) - -func TestSetAliasCAS_CreateFromEmpty(t *testing.T) { - repo := newTestRepo(t) - ctx := context.Background() - t0 := time.Now().UTC() - - cur, ok, err := repo.SetAliasCAS(ctx, "new-site", "production", "", "d1", t0) - require.NoError(t, err) - assert.True(t, ok, "CAS over an absent row with expected=\"\" creates the alias") - assert.Equal(t, "", cur, "current value of a fresh alias is empty") - - targets, _, err := repo.AliasTargets(ctx, "new-site") - require.NoError(t, err) - assert.Contains(t, targets, "d1", "the new alias points at the published deploy") - - events, err := repo.FetchUnpublished(ctx, 10) - require.NoError(t, err) - require.Len(t, events, 1, "first-publish enqueues exactly one outbox event") - assert.Equal(t, TopicSiteChanged, events[0].Topic) -} - -func TestSetAliasCAS_AbsentRowNonEmptyExpected(t *testing.T) { - repo := newTestRepo(t) - ctx := context.Background() - t0 := time.Now().UTC() - - cur, ok, err := repo.SetAliasCAS(ctx, "ghost-site", "production", "X", "d1", t0) - require.NoError(t, err) - assert.False(t, ok, "CAS over an absent row with a non-empty expected value is rejected") - assert.Equal(t, "", cur, "actual current value is empty for an absent row") - - targets, _, err := repo.AliasTargets(ctx, "ghost-site") - require.NoError(t, err) - assert.Empty(t, targets, "rejected CAS does not create the alias") - - events, err := repo.FetchUnpublished(ctx, 10) - require.NoError(t, err) - assert.Empty(t, events, "rejected CAS enqueues no outbox event") -} diff --git a/internal/pg/alias_test.go b/internal/pg/alias_test.go deleted file mode 100644 index ea306a0..0000000 --- a/internal/pg/alias_test.go +++ /dev/null @@ -1,64 +0,0 @@ -package pg - -import ( - "context" - "sync" - "testing" - "time" - - "github.com/stretchr/testify/assert" - "github.com/stretchr/testify/require" -) - -func TestAlias_NoLostUpdate(t *testing.T) { - repo := newTestRepo(t) - ctx := context.Background() - now := time.Now().UTC() - - require.NoError(t, repo.UpsertAlias(ctx, "www", "production", "A", now)) - - var wg sync.WaitGroup - type res struct { - ok bool - current string - err error - } - results := make([]res, 2) - nexts := []string{"B", "C"} - for i := 0; i < 2; i++ { - wg.Add(1) - go func(i int) { - defer wg.Done() - cur, ok, err := repo.SetAliasCAS(ctx, "www", "production", "A", nexts[i], now.Add(time.Minute)) - results[i] = res{ok, cur, err} - }(i) - } - wg.Wait() - - wins := 0 - for _, r := range results { - require.NoError(t, r.err) - if r.ok { - wins++ - } - } - assert.Equal(t, 1, wins, "exactly one concurrent CAS from the same expected value wins (V8 no lost update)") - - targets, _, err := repo.AliasTargets(ctx, "www") - require.NoError(t, err) - assert.Len(t, targets, 1, "alias holds a single, consistent value") - _, hasA := targets["A"] - assert.False(t, hasA, "the stale value was overwritten by the winner") -} - -func TestSetAliasCAS_DriftRejected(t *testing.T) { - repo := newTestRepo(t) - ctx := context.Background() - now := time.Now().UTC() - require.NoError(t, repo.UpsertAlias(ctx, "www", "production", "A", now)) - - cur, ok, err := repo.SetAliasCAS(ctx, "www", "production", "stale-expected", "Z", now) - require.NoError(t, err) - assert.False(t, ok, "CAS with wrong expected value is rejected") - assert.Equal(t, "A", cur, "caller is told the actual current value") -} diff --git a/internal/pg/migrate_test.go b/internal/pg/migrate_test.go index 104df7f..b2dfd1f 100644 --- a/internal/pg/migrate_test.go +++ b/internal/pg/migrate_test.go @@ -66,7 +66,7 @@ func TestMigrations(t *testing.T) { var indexDef string require.NoError(t, db.Pool.QueryRow(ctx, "SELECT indexdef FROM pg_indexes WHERE indexname = 'outbox_unpublished_idx'").Scan(&indexDef)) - require.Contains(t, indexDef, "(id)", "0004 rebuilt outbox_unpublished_idx on id to match FetchUnpublished ORDER BY id") + require.Contains(t, indexDef, "(id)", "0004 rebuilt outbox_unpublished_idx on id to match the relay claim ORDER BY id") require.NotContains(t, indexDef, "created_at", "stale created_at index dropped by 0004") var occurredIDIdx string @@ -87,10 +87,9 @@ func TestMigrations(t *testing.T) { repo := NewRepo(db) require.NoError(t, repo.EnqueueSiteChanged(ctx, "second")) require.NoError(t, repo.EnqueueSiteChanged(ctx, "third")) - events, err := repo.FetchUnpublished(ctx, 10) - require.NoError(t, err) + events := fetchUnpublished(ctx, t, repo, 10) require.Len(t, events, 2, "both enqueued events unpublished") - require.Less(t, events[0].ID, events[1].ID, "FetchUnpublished returns oldest-first by id") + require.Less(t, events[0].ID, events[1].ID, "unpublished events come back oldest-first by id") } func TestReleaseAdvisoryLock_FreesLockOnCanceledCallerCtx(t *testing.T) { diff --git a/internal/pg/outbox.go b/internal/pg/outbox.go index e0eb794..a8cc48a 100644 --- a/internal/pg/outbox.go +++ b/internal/pg/outbox.go @@ -40,28 +40,6 @@ func (r *Repo) EnqueueSiteChanged(ctx context.Context, site string) error { }) } -func (r *Repo) FetchUnpublished(ctx context.Context, limit int) ([]OutboxEvent, error) { - rows, err := r.pool.Query(ctx, - `SELECT id, topic, payload FROM outbox - WHERE published_at IS NULL - ORDER BY id - LIMIT $1`, limit) - if err != nil { - return nil, fmt.Errorf("pg outbox fetch: %w", err) - } - defer rows.Close() - - var out []OutboxEvent - for rows.Next() { - var e OutboxEvent - if err := rows.Scan(&e.ID, &e.Topic, &e.Payload); err != nil { - return nil, fmt.Errorf("pg outbox scan: %w", err) - } - out = append(out, e) - } - return out, rows.Err() -} - const claimTTL = 5 * time.Minute func (r *Repo) claimBatch(ctx context.Context, limit int) ([]OutboxEvent, error) { diff --git a/internal/pg/outbox_markpublished_test.go b/internal/pg/outbox_markpublished_test.go index fbc9e2a..144b75a 100644 --- a/internal/pg/outbox_markpublished_test.go +++ b/internal/pg/outbox_markpublished_test.go @@ -18,19 +18,16 @@ func TestMarkPublished_EmptyBatchMarksNothing(t *testing.T) { require.NoError(t, repo.MarkPublished(ctx, []int64{}, now), "empty id slice is a guarded no-op") require.NoError(t, repo.EnqueueSiteChanged(ctx, "www")) - events, err := repo.FetchUnpublished(ctx, 10) - require.NoError(t, err) + events := fetchUnpublished(ctx, t, repo, 10) require.Len(t, events, 1) require.NoError(t, repo.MarkPublished(ctx, nil, now), "a nil batch must not touch existing unpublished rows") - still, err := repo.FetchUnpublished(ctx, 10) - require.NoError(t, err) + still := fetchUnpublished(ctx, t, repo, 10) require.Len(t, still, 1, "the empty-batch no-op left the real event unpublished") ids := []int64{events[0].ID} require.NoError(t, repo.MarkPublished(ctx, ids, now)) - after, err := repo.FetchUnpublished(ctx, 10) - require.NoError(t, err) + after := fetchUnpublished(ctx, t, repo, 10) assert.Empty(t, after, "a non-empty batch marks the event published") } diff --git a/internal/pg/outbox_relaybatch_test.go b/internal/pg/outbox_relaybatch_test.go index 366a375..ae173c5 100644 --- a/internal/pg/outbox_relaybatch_test.go +++ b/internal/pg/outbox_relaybatch_test.go @@ -61,8 +61,7 @@ func TestRelayBatch_ExclusiveAcrossReplicas(t *testing.T) { assert.Equal(t, 1, c, "event %d must publish exactly once across %d replicas while its claim marker holds (B3); duplicates mean the claim is not exclusive", id, replicas) } - remaining, err := repo.FetchUnpublished(ctx, total) - require.NoError(t, err) + remaining := fetchUnpublished(ctx, t, repo, total) assert.Empty(t, remaining, "every claimed event eventually marked published") } @@ -86,8 +85,7 @@ func TestRelayBatch_PublishFailureLeavesEventUnpublished(t *testing.T) { require.Error(t, err) assert.Equal(t, 1, n, "only the pre-failure event marked") - remaining, err := repo.FetchUnpublished(ctx, 10) - require.NoError(t, err) + remaining := fetchUnpublished(ctx, t, repo, 10) require.Len(t, remaining, 1, "the failed event stays unpublished for retry (at-least-once)") assert.Equal(t, "b", payloadSite(t, remaining[0])) } @@ -121,8 +119,7 @@ func TestRelayBatch_MarkSurvivesContextDeath(t *testing.T) { require.Error(t, err, "the publish failure must surface, not be swallowed by the mark") assert.Equal(t, 1, n, "the pre-failure publish must be marked even though the batch ctx died") - remaining, ferr := repo.FetchUnpublished(context.Background(), 10) - require.NoError(t, ferr) + remaining := fetchUnpublished(context.Background(), t, repo, 10) assert.Len(t, remaining, 2, "only the published event may be marked; the rest stay for retry") } diff --git a/internal/pg/outbox_test.go b/internal/pg/outbox_test.go index e929ae4..f7ec563 100644 --- a/internal/pg/outbox_test.go +++ b/internal/pg/outbox_test.go @@ -12,6 +12,26 @@ import ( "github.com/stretchr/testify/require" ) +func fetchUnpublished(ctx context.Context, t *testing.T, repo *Repo, limit int) []OutboxEvent { + t.Helper() + rows, err := repo.pool.Query(ctx, + `SELECT id, topic, payload FROM outbox + WHERE published_at IS NULL + ORDER BY id + LIMIT $1`, limit) + require.NoError(t, err) + defer rows.Close() + + var out []OutboxEvent + for rows.Next() { + var e OutboxEvent + require.NoError(t, rows.Scan(&e.ID, &e.Topic, &e.Payload)) + out = append(out, e) + } + require.NoError(t, rows.Err()) + return out +} + func TestOutbox_AtomicWithMetadataAndRelay(t *testing.T) { repo := newTestRepo(t) ctx := context.Background() @@ -44,8 +64,7 @@ func TestOutbox_AtomicWithMetadataAndRelay(t *testing.T) { assert.True(t, ids["d1"], "committed metadata present") assert.False(t, ids["d2"], "rolled-back metadata absent (dual-write closed)") - events, err := repo.FetchUnpublished(ctx, 10) - require.NoError(t, err) + events := fetchUnpublished(ctx, t, repo, 10) require.Len(t, events, 1, "only the committed tx produced an outbox row") assert.Equal(t, TopicSiteChanged, events[0].Topic) var p map[string]string @@ -53,8 +72,7 @@ func TestOutbox_AtomicWithMetadataAndRelay(t *testing.T) { assert.Equal(t, "www", p["site"]) require.NoError(t, repo.MarkPublished(ctx, []int64{events[0].ID}, time.Now())) - again, err := repo.FetchUnpublished(ctx, 10) - require.NoError(t, err) + again := fetchUnpublished(ctx, t, repo, 10) assert.Empty(t, again, "published events are not re-fetched") } @@ -63,8 +81,7 @@ func TestRelayBatch_CommitsClaimBeforePublish(t *testing.T) { ctx := context.Background() require.NoError(t, repo.EnqueueSiteChanged(ctx, "www")) - events, err := repo.FetchUnpublished(ctx, 10) - require.NoError(t, err) + events := fetchUnpublished(ctx, t, repo, 10) require.Len(t, events, 1) id := events[0].ID @@ -95,8 +112,7 @@ func TestOutbox_EnqueueSiteChanged(t *testing.T) { ctx := context.Background() require.NoError(t, repo.EnqueueSiteChanged(ctx, "learn")) - events, err := repo.FetchUnpublished(ctx, 10) - require.NoError(t, err) + events := fetchUnpublished(ctx, t, repo, 10) require.Len(t, events, 1) var p map[string]string require.NoError(t, json.Unmarshal(events[0].Payload, &p)) diff --git a/internal/pg/saga_test.go b/internal/pg/saga_test.go index f023b47..e9dce25 100644 --- a/internal/pg/saga_test.go +++ b/internal/pg/saga_test.go @@ -26,8 +26,7 @@ func TestDeploySaga(t *testing.T) { require.NoError(t, err) assert.Contains(t, targets, "20260420-141522-abc1234", "production alias points at the finalized deploy") - events, err := repo.FetchUnpublished(ctx, 10) - require.NoError(t, err) + events := fetchUnpublished(ctx, t, repo, 10) require.Len(t, events, 1, "exactly one site.changed emitted in the same tx") assert.Equal(t, TopicSiteChanged, events[0].Topic) diff --git a/internal/server/server.go b/internal/server/server.go index 8e03870..92367ed 100644 --- a/internal/server/server.go +++ b/internal/server/server.go @@ -26,6 +26,7 @@ // GET /api/repo/{id} — GitHub bearer (feature-gated) // POST /api/repo/{id}/approve — GitHub bearer + repo-approve team (feature-gated) // POST /api/repo/{id}/reject — GitHub bearer + repo-approve team (feature-gated) +// DELETE /api/repo/{id} — GitHub bearer + repo-approve team (feature-gated) package server import ( diff --git a/internal/teamcache/teamcache.go b/internal/teamcache/teamcache.go index 6407b0b..78cc0db 100644 --- a/internal/teamcache/teamcache.go +++ b/internal/teamcache/teamcache.go @@ -51,19 +51,3 @@ func (c *Cache) Set(ctx context.Context, login string, teams []string) error { } return nil } - -func (c *Cache) GetOrFetch(ctx context.Context, login string, fetch func(ctx context.Context) ([]string, error)) ([]string, error) { - if teams, hit, err := c.Get(ctx, login); err != nil { - return nil, err - } else if hit { - return teams, nil - } - teams, err := fetch(ctx) - if err != nil { - return nil, err - } - if err := c.Set(ctx, login, teams); err != nil { - return nil, err - } - return teams, nil -} diff --git a/internal/teamcache/teamcache_errors_test.go b/internal/teamcache/teamcache_errors_test.go deleted file mode 100644 index fece72a..0000000 --- a/internal/teamcache/teamcache_errors_test.go +++ /dev/null @@ -1,25 +0,0 @@ -package teamcache - -import ( - "context" - "testing" - "time" - - "github.com/stretchr/testify/assert" - "github.com/stretchr/testify/require" -) - -func TestTeamCache_GetOrFetch_AbortsOnGetError(t *testing.T) { - ctx := context.Background() - c, mr := newTestCache(t, time.Minute) - mr.SetError("valkey down") - - calls := 0 - _, err := c.GetOrFetch(ctx, "b", func(context.Context) ([]string, error) { - calls++ - return nil, nil - }) - require.Error(t, err) - assert.Equal(t, 0, calls, "fetch must not run when the cache Get itself errors") - assert.ErrorContains(t, err, "teamcache get b") -} diff --git a/internal/teamcache/teamcache_test.go b/internal/teamcache/teamcache_test.go index d24e5be..500b2b9 100644 --- a/internal/teamcache/teamcache_test.go +++ b/internal/teamcache/teamcache_test.go @@ -2,7 +2,6 @@ package teamcache import ( "context" - "errors" "testing" "time" @@ -54,49 +53,6 @@ func TestTeamCache(t *testing.T) { assert.Equal(t, []string{"staff", "team-eng"}, teams) } -func TestTeamCache_GetOrFetch_FetchesOnceThenCaches(t *testing.T) { - ctx := context.Background() - c, _ := newTestCache(t, 5*time.Minute) - - calls := 0 - fetch := func(context.Context) ([]string, error) { - calls++ - return []string{"staff"}, nil - } - - teams, err := c.GetOrFetch(ctx, "bob", fetch) - require.NoError(t, err) - assert.Equal(t, []string{"staff"}, teams) - assert.Equal(t, 1, calls, "miss triggers exactly one upstream fetch") - - teams, err = c.GetOrFetch(ctx, "bob", fetch) - require.NoError(t, err) - assert.Equal(t, []string{"staff"}, teams) - assert.Equal(t, 1, calls, "second call served from Valkey cache; GitHub App quota protected") -} - -func TestTeamCache_CachesEmptyMembership(t *testing.T) { - ctx := context.Background() - c, _ := newTestCache(t, 5*time.Minute) - - calls := 0 - fetch := func(context.Context) ([]string, error) { - calls++ - return nil, nil - } - _, err := c.GetOrFetch(ctx, "outsider", fetch) - require.NoError(t, err) - - teams, hit, err := c.Get(ctx, "outsider") - require.NoError(t, err) - assert.True(t, hit, "an empty team list is cached, not treated as a miss") - assert.Empty(t, teams) - - _, err = c.GetOrFetch(ctx, "outsider", fetch) - require.NoError(t, err) - assert.Equal(t, 1, calls, "non-member result is cached too — no re-fetch storm") -} - func TestTeamCache_TTLExpiry(t *testing.T) { ctx := context.Background() c, mr := newTestCache(t, time.Minute) @@ -109,20 +65,6 @@ func TestTeamCache_TTLExpiry(t *testing.T) { assert.False(t, hit, "entry expires after TTL -> miss") } -func TestTeamCache_FetchErrorNotCached(t *testing.T) { - ctx := context.Background() - c, _ := newTestCache(t, time.Minute) - - _, err := c.GetOrFetch(ctx, "carol", func(context.Context) ([]string, error) { - return nil, errors.New("github 503") - }) - require.Error(t, err) - - _, hit, err := c.Get(ctx, "carol") - require.NoError(t, err) - assert.False(t, hit, "a failed upstream fetch is never cached") -} - func TestTeamCache_Get_MalformedJSONIsAnError(t *testing.T) { ctx := context.Background() c, mr := newTestCache(t, time.Minute) @@ -146,19 +88,3 @@ func TestTeamCache_Get_RedisErrorPropagates(t *testing.T) { assert.Nil(t, teams) assert.ErrorContains(t, err, "teamcache get") } - -func TestTeamCache_GetOrFetch_SetFailurePropagates(t *testing.T) { - ctx := context.Background() - c, mr := newTestCache(t, time.Minute) - failCommands(t, mr, "READONLY You can't write against a read only replica.", "SET") - - calls := 0 - teams, err := c.GetOrFetch(ctx, "bob", func(context.Context) ([]string, error) { - calls++ - return []string{"staff"}, nil - }) - require.Error(t, err, "an unpersisted fetch must surface the write error, not pose as cached") - assert.Nil(t, teams) - assert.Equal(t, 1, calls, "the miss path runs fetch before the failing Set") - assert.ErrorContains(t, err, "teamcache set") -} diff --git a/internal/worker/debounce.go b/internal/worker/debounce.go deleted file mode 100644 index e465c0b..0000000 --- a/internal/worker/debounce.go +++ /dev/null @@ -1,62 +0,0 @@ -package worker - -import ( - "sync" - "time" -) - -type Debouncer struct { - Window time.Duration - Trigger func(site string) - - mu sync.Mutex - gen uint64 - timers map[string]debounceEntry - stopped bool -} - -type debounceEntry struct { - timer *time.Timer - gen uint64 -} - -func NewDebouncer(window time.Duration, trigger func(site string)) *Debouncer { - return &Debouncer{Window: window, Trigger: trigger, timers: map[string]debounceEntry{}} -} - -func (d *Debouncer) Notify(site string) { - d.mu.Lock() - defer d.mu.Unlock() - if d.stopped { - return - } - if e, ok := d.timers[site]; ok { - e.timer.Stop() - } - d.gen++ - gen := d.gen - timer := time.AfterFunc(d.Window, func() { d.fire(site, gen) }) - d.timers[site] = debounceEntry{timer: timer, gen: gen} -} - -func (d *Debouncer) fire(site string, gen uint64) { - d.mu.Lock() - e, ok := d.timers[site] - if d.stopped || !ok || e.gen != gen { - d.mu.Unlock() - return - } - delete(d.timers, site) - d.mu.Unlock() - d.Trigger(site) -} - -func (d *Debouncer) Stop() { - d.mu.Lock() - defer d.mu.Unlock() - d.stopped = true - for _, e := range d.timers { - e.timer.Stop() - } - d.timers = map[string]debounceEntry{} -} diff --git a/internal/worker/debounce_test.go b/internal/worker/debounce_test.go deleted file mode 100644 index a97e0ea..0000000 --- a/internal/worker/debounce_test.go +++ /dev/null @@ -1,103 +0,0 @@ -package worker - -import ( - "sync" - "testing" - "time" - - "github.com/stretchr/testify/assert" - "github.com/stretchr/testify/require" -) - -func TestDebounce(t *testing.T) { - var mu sync.Mutex - fired := map[string]int{} - d := NewDebouncer(30*time.Millisecond, func(site string) { - mu.Lock() - fired[site]++ - mu.Unlock() - }) - t.Cleanup(d.Stop) - - for i := 0; i < 5; i++ { - d.Notify("www") - } - d.Notify("learn") - - require.Eventually(t, func() bool { - mu.Lock() - defer mu.Unlock() - return fired["www"] == 1 && fired["learn"] == 1 - }, time.Second, 5*time.Millisecond, "a burst per site coalesces into exactly one trigger") - - mu.Lock() - assert.Equal(t, 1, fired["www"], "5 rapid site.changed events -> 1 gc-site trigger") - mu.Unlock() -} - -func TestDebounce_LaterChangeRetriggers(t *testing.T) { - var mu sync.Mutex - var count int - d := NewDebouncer(20*time.Millisecond, func(string) { - mu.Lock() - count++ - mu.Unlock() - }) - t.Cleanup(d.Stop) - - d.Notify("www") - require.Eventually(t, func() bool { mu.Lock(); defer mu.Unlock(); return count == 1 }, time.Second, 5*time.Millisecond) - - d.Notify("www") - require.Eventually(t, func() bool { mu.Lock(); defer mu.Unlock(); return count == 2 }, time.Second, 5*time.Millisecond, - "a change after processing triggers GC again (no lost updates; per-site order preserved by engine key, E2)") -} - -func TestDebounce_StaleCallbackDoesNotDropNewerTimer(t *testing.T) { - var mu sync.Mutex - var count int - d := NewDebouncer(time.Hour, func(string) { - mu.Lock() - count++ - mu.Unlock() - }) - t.Cleanup(d.Stop) - - d.Notify("www") - d.mu.Lock() - stale := d.timers["www"] - d.mu.Unlock() - - d.Notify("www") - d.mu.Lock() - fresh := d.timers["www"] - d.mu.Unlock() - require.NotEqual(t, stale.gen, fresh.gen, "second Notify installs a distinct timer") - - d.fire("www", stale.gen) - - mu.Lock() - assert.Equal(t, 0, count, "stale in-flight callback must not Trigger") - mu.Unlock() - - d.mu.Lock() - got := d.timers["www"] - d.mu.Unlock() - assert.Equal(t, fresh.gen, got.gen, "stale callback must not delete the newer timer entry") -} - -func TestDebounce_StopHaltsPendingTriggers(t *testing.T) { - var mu sync.Mutex - var count int - d := NewDebouncer(50*time.Millisecond, func(string) { - mu.Lock() - count++ - mu.Unlock() - }) - d.Notify("www") - d.Stop() - time.Sleep(80 * time.Millisecond) - mu.Lock() - assert.Equal(t, 0, count, "Stop cancels pending triggers") - mu.Unlock() -} diff --git a/internal/worker/deployflows.go b/internal/worker/deployflows.go deleted file mode 100644 index 8cf1b43..0000000 --- a/internal/worker/deployflows.go +++ /dev/null @@ -1,21 +0,0 @@ -package worker - -const ( - WorkflowFinalize = "finalize" - WorkflowPromote = "promote" - WorkflowRollback = "rollback" -) - -func RegisterDeployWorkflows(rt *Runtime, finalize, promote, rollback Handler) error { - defs := []WorkflowDef{ - {Name: WorkflowFinalize, ConcurrencyKey: ConcurrencyKeySite, Handler: finalize}, - {Name: WorkflowPromote, ConcurrencyKey: ConcurrencyKeySite, Handler: promote}, - {Name: WorkflowRollback, ConcurrencyKey: ConcurrencyKeySite, Handler: rollback}, - } - for _, d := range defs { - if err := rt.Register(d); err != nil { - return err - } - } - return nil -} diff --git a/internal/worker/deployflows_test.go b/internal/worker/deployflows_test.go deleted file mode 100644 index 49ffd9a..0000000 --- a/internal/worker/deployflows_test.go +++ /dev/null @@ -1,32 +0,0 @@ -package worker - -import ( - "testing" - - "github.com/stretchr/testify/assert" - "github.com/stretchr/testify/require" -) - -func TestDeployWorkflowsRegisterWithSiteKey(t *testing.T) { - eng := &fakeEngine{} - rt := NewRuntime(eng) - - require.NoError(t, RegisterDeployWorkflows(rt, noop, noop, noop)) - - byName := map[string]WorkflowDef{} - for _, d := range eng.registered { - byName[d.Name] = d - } - require.Len(t, eng.registered, 3) - for _, name := range []string{WorkflowFinalize, WorkflowPromote, WorkflowRollback} { - assert.Equal(t, ConcurrencyKeySite, byName[name].ConcurrencyKey, - "%s must serialize per-site via concurrency key (V8 single-writer-per-site)", name) - } -} - -func TestRegisterDeployWorkflows_PropagatesError(t *testing.T) { - rt := NewRuntime(&fakeEngine{}) - require.NoError(t, RegisterDeployWorkflows(rt, noop, noop, noop)) - err := RegisterDeployWorkflows(rt, noop, noop, noop) - require.Error(t, err, "re-registering the same workflow names is rejected") -} diff --git a/internal/worker/runtime.go b/internal/worker/runtime.go index 349b216..9f5dd72 100644 --- a/internal/worker/runtime.go +++ b/internal/worker/runtime.go @@ -11,10 +11,8 @@ const ConcurrencyKeySite = "site" const ( WorkflowGCSite = "gc-site" - WorkflowManualDelete = "manual-delete" - WorkflowSitePurge = "site-purge" WorkflowTombstonePurge = "tombstone-purge" - WorkflowReconcile = "reconcile" + WorkflowDriftDetect = "drift-detect" ) type Handler func(ctx context.Context, input map[string]any) error diff --git a/internal/worker/runtime_test.go b/internal/worker/runtime_test.go index dc89853..cfb0802 100644 --- a/internal/worker/runtime_test.go +++ b/internal/worker/runtime_test.go @@ -33,7 +33,7 @@ func TestWorkerBoot(t *testing.T) { eng := &fakeEngine{} rt := NewRuntime(eng) - perSite := []string{WorkflowGCSite, WorkflowManualDelete, WorkflowSitePurge} + perSite := []string{WorkflowGCSite, "manual-delete", "site-purge"} for _, name := range perSite { require.NoError(t, rt.Register(WorkflowDef{Name: name, ConcurrencyKey: ConcurrencyKeySite, Handler: noop})) } From fc0e5b65da940be14d0270f65e47115963286ba7 Mon Sep 17 00:00:00 2001 From: Mrugesh Mohapatra Date: Mon, 17 Aug 2026 18:08:00 +0530 Subject: [PATCH 33/41] chore(repo): drop dead valkey store, fix op tags --- internal/handler/repo.go | 20 +- internal/reporequest/valkey/store.go | 402 --------------------- internal/reporequest/valkey/store_test.go | 404 ---------------------- 3 files changed, 10 insertions(+), 816 deletions(-) delete mode 100644 internal/reporequest/valkey/store.go delete mode 100644 internal/reporequest/valkey/store_test.go diff --git a/internal/handler/repo.go b/internal/handler/repo.go index 9b8298e..c892477 100644 --- a/internal/handler/repo.go +++ b/internal/handler/repo.go @@ -182,7 +182,7 @@ func (h *Handlers) RepoCreate(w http.ResponseWriter, r *http.Request) { "a request for this repo name is already pending or active") return } - writeUpstreamError(w, r, http.StatusBadGateway, "repo_store_failed", "valkey.repo.create", err) + writeUpstreamError(w, r, http.StatusBadGateway, "repo_store_failed", "pg.repo.create", err) return } slog.InfoContext(r.Context(), "repo.create.queued", "id", created.ID, "name", req.Name, "owner", h.RepoOrg, "visibility", string(vis)) @@ -244,7 +244,7 @@ func (h *Handlers) ReposList(w http.ResponseWriter, r *http.Request) { all, err := h.Repos.List(r.Context()) if err != nil { - writeUpstreamError(w, r, http.StatusBadGateway, "repo_store_failed", "valkey.repo.list", err) + writeUpstreamError(w, r, http.StatusBadGateway, "repo_store_failed", "pg.repo.list", err) return } seesActors := h.callerSeesActors(r) @@ -278,7 +278,7 @@ func (h *Handlers) RepoGet(w http.ResponseWriter, r *http.Request) { writeError(w, http.StatusNotFound, "not_found", "repo request not found") return } - writeUpstreamError(w, r, http.StatusBadGateway, "repo_store_failed", "valkey.repo.get", err) + writeUpstreamError(w, r, http.StatusBadGateway, "repo_store_failed", "pg.repo.get", err) return } row := toRepoRow(req) @@ -323,7 +323,7 @@ func (h *Handlers) RepoApprove(w http.ResponseWriter, r *http.Request) { // reconcile; otherwise it is genuinely resolved (PR freeCodeCamp/artemis#3, #7). cur, gErr := h.Repos.Get(r.Context(), id) if gErr != nil { - writeUpstreamError(w, r, http.StatusBadGateway, "repo_store_failed", "valkey.repo.get", gErr) + writeUpstreamError(w, r, http.StatusBadGateway, "repo_store_failed", "pg.repo.get", gErr) return } if cur.Status != reporequest.StatusApproved { @@ -334,7 +334,7 @@ func (h *Handlers) RepoApprove(w http.ResponseWriter, r *http.Request) { approved, resume = cur, true slog.WarnContext(r.Context(), "repo.approve.resume_stranded_approved", "id", id) default: - writeUpstreamError(w, r, http.StatusBadGateway, "repo_store_failed", "valkey.repo.approve", err) + writeUpstreamError(w, r, http.StatusBadGateway, "repo_store_failed", "pg.repo.approve", err) return } @@ -390,7 +390,7 @@ func (h *Handlers) RepoApprove(w http.ResponseWriter, r *http.Request) { } failed, mErr := h.Repos.MarkFailed(durCtx, id, msg) if mErr != nil { - writeUpstreamError(w, r, http.StatusBadGateway, "repo_store_failed", "valkey.repo.markfailed", mErr) + writeUpstreamError(w, r, http.StatusBadGateway, "repo_store_failed", "pg.repo.markfailed", mErr) return } h.auditFromScope(durCtx, "repo.approve", "approved_failed", map[string]any{"id": id, "name": approved.Name}) @@ -400,7 +400,7 @@ func (h *Handlers) RepoApprove(w http.ResponseWriter, r *http.Request) { active, mErr := h.Repos.MarkActive(durCtx, id, created.URL) if mErr != nil { - writeUpstreamError(w, r, http.StatusBadGateway, "repo_store_failed", "valkey.repo.markactive", mErr) + writeUpstreamError(w, r, http.StatusBadGateway, "repo_store_failed", "pg.repo.markactive", mErr) return } slog.InfoContext(r.Context(), "repo.approve.created", "id", id, "name", active.Name, "url", created.URL) @@ -445,7 +445,7 @@ func (h *Handlers) RepoReject(w http.ResponseWriter, r *http.Request) { writeError(w, http.StatusConflict, "already_resolved", "request was already resolved by another admin") default: - writeUpstreamError(w, r, http.StatusBadGateway, "repo_store_failed", "valkey.repo.reject", err) + writeUpstreamError(w, r, http.StatusBadGateway, "repo_store_failed", "pg.repo.reject", err) } return } @@ -465,7 +465,7 @@ func (h *Handlers) RepoDelete(w http.ResponseWriter, r *http.Request) { writeError(w, http.StatusNotFound, "not_found", "repo request not found") return } - writeUpstreamError(w, r, http.StatusBadGateway, "repo_store_failed", "valkey.repo.get", err) + writeUpstreamError(w, r, http.StatusBadGateway, "repo_store_failed", "pg.repo.get", err) return } if err := h.Repos.Delete(r.Context(), id); err != nil { @@ -473,7 +473,7 @@ func (h *Handlers) RepoDelete(w http.ResponseWriter, r *http.Request) { writeError(w, http.StatusNotFound, "not_found", "repo request not found") return } - writeUpstreamError(w, r, http.StatusBadGateway, "repo_store_failed", "valkey.repo.delete", err) + writeUpstreamError(w, r, http.StatusBadGateway, "repo_store_failed", "pg.repo.delete", err) return } slog.InfoContext(r.Context(), "repo.delete.removed", "id", id, "name", row.Name) diff --git a/internal/reporequest/valkey/store.go b/internal/reporequest/valkey/store.go deleted file mode 100644 index 73e2cfd..0000000 --- a/internal/reporequest/valkey/store.go +++ /dev/null @@ -1,402 +0,0 @@ -// Package valkey is the Valkey-backed implementation of the repo-request -// queue. It mirrors the site-registry store's WATCH/MULTI optimistic- -// transaction discipline so concurrent admins can never double-resolve a -// request: the approve/reject transition is a compare-and-set guarded on -// the row's pending status (dossier §V3). -// -// Wire schema: -// -// - HSET repo: — hash row per request -// - SADD repos:all — index set of every request id -// - SADD repos:names — set of repo names currently claimed (a name -// is claimed while pending/approved/active; released on reject/fail) -package valkey - -import ( - "context" - "crypto/rand" - "encoding/hex" - "errors" - "fmt" - "sort" - "strings" - "time" - - "github.com/redis/go-redis/v9" - - "github.com/freeCodeCamp/artemis/internal/reporequest" -) - -const ( - keyAllRequests = "repos:all" - keyClaimedNames = "repos:names" -) - -const ( - fieldName = "name" - fieldOwner = "owner" - fieldVisibility = "visibility" - fieldDescription = "description" - fieldTemplate = "template" - fieldStatus = "status" - fieldURL = "url" - fieldError = "error" - fieldRequestedBy = "requested_by" - fieldApprover = "approver" - fieldRejectReason = "reject_reason" - fieldCreatedAt = "created_at" - fieldUpdatedAt = "updated_at" -) - -// Re-exports so callers in this package stay decoupled from the domain -// import path. -type Request = reporequest.Request - -var ( - ErrNotFound = reporequest.ErrNotFound - ErrAlreadyExists = reporequest.ErrAlreadyExists - ErrNotPending = reporequest.ErrNotPending -) - -// Config carries the Valkey connection details. -type Config struct { - Addr string - Password string -} - -// Store is the repo-request queue backed by a go-redis client. -type Store struct { - client *redis.Client - - // Now stamps created_at / updated_at. Tests inject a deterministic - // clock; production uses time.Now. - Now func() time.Time - // NewID mints request ids. Tests inject a deterministic generator; - // production uses a crypto/rand-backed "req_". - NewID func() string -} - -// New dials Valkey, verifies connectivity, and returns a ready Store. -func New(ctx context.Context, cfg Config) (*Store, error) { - if cfg.Addr == "" { - return nil, errors.New("reporequest/valkey: empty Addr") - } - c := redis.NewClient(&redis.Options{Addr: cfg.Addr, Password: cfg.Password}) - if err := c.Ping(ctx).Err(); err != nil { - _ = c.Close() - return nil, fmt.Errorf("reporequest/valkey: ping %s: %w", cfg.Addr, err) - } - return &Store{client: c, Now: time.Now, NewID: defaultNewID}, nil -} - -// NewWithClient wraps an existing go-redis client (lets main.go share one -// connection pool across the site registry and repo-request stores). -// Returns an error on a nil client rather than deferring to a nil-pointer -// panic on the first store call. -func NewWithClient(c *redis.Client) (*Store, error) { - if c == nil { - return nil, errors.New("reporequest/valkey: nil client") - } - return &Store{client: c, Now: time.Now, NewID: defaultNewID}, nil -} - -// Ping verifies the connection. Cheap; safe on a liveness probe. -func (s *Store) Ping(ctx context.Context) error { return s.client.Ping(ctx).Err() } - -// Close releases the connection pool. -func (s *Store) Close() error { return s.client.Close() } - -func defaultNewID() string { - var b [10]byte - _, _ = rand.Read(b[:]) - return "req_" + hex.EncodeToString(b[:]) -} - -func reqKey(id string) string { return "repo:" + id } - -// nameClaimKey normalizes a repo name for the dedupe claim set. GitHub -// repo names are case-insensitive for uniqueness, so "MyRepo" and -// "myrepo" must collide in the queue — claim on the lowercased name -// (the row still stores the requester's original casing for creation). -func nameClaimKey(name string) string { return strings.ToLower(name) } - -// Create writes a new pending request. Returns ErrAlreadyExists if a -// request for the same repo name is already claimed (pending/approved/ -// active). The name-claim check + writes run in one optimistic -// transaction so concurrent submits of the same name resolve to exactly -// one winner. -func (s *Store) Create(ctx context.Context, req Request) (Request, error) { - if req.Name == "" { - return Request{}, errors.New("reporequest/valkey: empty name") - } - now := s.Now().UTC() - req.ID = s.NewID() - req.Status = reporequest.StatusPending - req.CreatedAt = now - req.UpdatedAt = now - - txf := func(tx *redis.Tx) error { - claimed, err := tx.SIsMember(ctx, keyClaimedNames, nameClaimKey(req.Name)).Result() - if err != nil { - return err - } - if claimed { - return ErrAlreadyExists - } - _, err = tx.TxPipelined(ctx, func(pipe redis.Pipeliner) error { - pipe.HSet(ctx, reqKey(req.ID), encodeFields(req)...) - pipe.SAdd(ctx, keyAllRequests, req.ID) - pipe.SAdd(ctx, keyClaimedNames, nameClaimKey(req.Name)) - return nil - }) - return err - } - - if err := s.watch(ctx, txf, keyClaimedNames); err != nil { - return Request{}, err - } - return req, nil -} - -// Get returns a single request or ErrNotFound. -func (s *Store) Get(ctx context.Context, id string) (Request, error) { - vals, err := s.client.HGetAll(ctx, reqKey(id)).Result() - if err != nil { - return Request{}, err - } - if len(vals) == 0 { - return Request{}, ErrNotFound - } - return decodeRequest(id, vals) -} - -// List returns every request sorted by created_at ascending (id as tie- -// breaker). Filtering by status / requester is the handler's concern. -func (s *Store) List(ctx context.Context) ([]Request, error) { - ids, err := s.client.SMembers(ctx, keyAllRequests).Result() - if err != nil { - return nil, err - } - out := make([]Request, 0, len(ids)) - for _, id := range ids { - vals, err := s.client.HGetAll(ctx, reqKey(id)).Result() - if err != nil { - return nil, err - } - if len(vals) == 0 { - continue // deleted out-of-band between SMEMBERS and HGETALL - } - req, err := decodeRequest(id, vals) - if err != nil { - return nil, err - } - out = append(out, req) - } - sort.Slice(out, func(i, j int) bool { - if out[i].CreatedAt.Equal(out[j].CreatedAt) { - return out[i].ID < out[j].ID - } - return out[i].CreatedAt.Before(out[j].CreatedAt) - }) - return out, nil -} - -// Approve flips a pending request to approved and records the approver. -// CAS guard: a request that is no longer pending returns ErrNotPending, -// so only one of several racing admins wins (dossier §V3). The name -// remains claimed (the repo is about to be created). -func (s *Store) Approve(ctx context.Context, id, approver string) (Request, error) { - return s.mutate(ctx, id, func(cur Request) (Request, bool, error) { - if !cur.Status.CanResolve() { - return Request{}, false, ErrNotPending - } - cur.Status = reporequest.StatusApproved - cur.Approver = approver - return cur, false, nil - }) -} - -// Reject flips a pending request to rejected, records the approver + -// reason, and releases the name claim. CAS-guarded like Approve. -func (s *Store) Reject(ctx context.Context, id, approver, reason string) (Request, error) { - return s.mutate(ctx, id, func(cur Request) (Request, bool, error) { - if !cur.Status.CanResolve() { - return Request{}, false, ErrNotPending - } - cur.Status = reporequest.StatusRejected - cur.Approver = approver - cur.RejectReason = reason - return cur, true, nil // release name - }) -} - -// MarkActive records a successful repo creation: approved → active with -// the repo URL. The name stays claimed (the repo now exists). -func (s *Store) MarkActive(ctx context.Context, id, url string) (Request, error) { - return s.mutate(ctx, id, func(cur Request) (Request, bool, error) { - if cur.Status != reporequest.StatusApproved { - return Request{}, false, ErrNotPending - } - cur.Status = reporequest.StatusActive - cur.URL = url - return cur, false, nil - }) -} - -// MarkFailed records a failed repo creation: approved → failed with the -// error message, releasing the name so the request can be retried. -func (s *Store) MarkFailed(ctx context.Context, id, errMsg string) (Request, error) { - return s.mutate(ctx, id, func(cur Request) (Request, bool, error) { - if cur.Status != reporequest.StatusApproved { - return Request{}, false, ErrNotPending - } - cur.Status = reporequest.StatusFailed - cur.Error = errMsg - return cur, true, nil // release name - }) -} - -func (s *Store) Delete(ctx context.Context, id string) error { - txf := func(tx *redis.Tx) error { - vals, err := tx.HGetAll(ctx, reqKey(id)).Result() - if err != nil { - return err - } - if len(vals) == 0 { - return ErrNotFound - } - cur, err := decodeRequest(id, vals) - if err != nil { - return err - } - _, err = tx.TxPipelined(ctx, func(pipe redis.Pipeliner) error { - pipe.Del(ctx, reqKey(id)) - pipe.SRem(ctx, keyAllRequests, id) - if cur.Status.HoldsName() { - pipe.SRem(ctx, keyClaimedNames, nameClaimKey(cur.Name)) - } - return nil - }) - return err - } - return s.watch(ctx, txf, reqKey(id), keyClaimedNames) -} - -func (s *Store) MarkStale(ctx context.Context, id, reason string) (Request, error) { - return s.mutate(ctx, id, func(cur Request) (Request, bool, error) { - if cur.Status != reporequest.StatusActive { - return Request{}, false, reporequest.ErrNotActive - } - cur.Status = reporequest.StatusFailed - cur.Error = reason - return cur, true, nil - }) -} - -// mutate applies fn to the current row inside a WATCH/MULTI transaction. -// fn returns the next row, whether to release the name claim, and an -// error to abort. updated_at is stamped automatically. -func (s *Store) mutate(ctx context.Context, id string, fn func(Request) (Request, bool, error)) (Request, error) { - var result Request - txf := func(tx *redis.Tx) error { - vals, err := tx.HGetAll(ctx, reqKey(id)).Result() - if err != nil { - return err - } - if len(vals) == 0 { - return ErrNotFound - } - cur, err := decodeRequest(id, vals) - if err != nil { - return err - } - next, release, err := fn(cur) - if err != nil { - return err - } - next.UpdatedAt = s.Now().UTC() - _, err = tx.TxPipelined(ctx, func(pipe redis.Pipeliner) error { - pipe.HSet(ctx, reqKey(id), encodeFields(next)...) - if release { - pipe.SRem(ctx, keyClaimedNames, nameClaimKey(next.Name)) - } - return nil - }) - if err != nil { - return err - } - result = next - return nil - } - - if err := s.watch(ctx, txf, reqKey(id)); err != nil { - return Request{}, err - } - return result, nil -} - -// watch runs txf under optimistic locking on the given keys, retrying on -// the redis.TxFailedErr optimistic-lock conflict. -func (s *Store) watch(ctx context.Context, txf func(*redis.Tx) error, keys ...string) error { - for { - err := s.client.Watch(ctx, txf, keys...) - switch { - case err == nil: - return nil - case errors.Is(err, redis.TxFailedErr): - continue - default: - return err - } - } -} - -func encodeFields(r Request) []any { - return []any{ - fieldName, r.Name, - fieldOwner, r.Owner, - fieldVisibility, string(r.Visibility), - fieldDescription, r.Description, - fieldTemplate, r.Template, - fieldStatus, string(r.Status), - fieldURL, r.URL, - fieldError, r.Error, - fieldRequestedBy, r.RequestedBy, - fieldApprover, r.Approver, - fieldRejectReason, r.RejectReason, - fieldCreatedAt, r.CreatedAt.Format(time.RFC3339Nano), - fieldUpdatedAt, r.UpdatedAt.Format(time.RFC3339Nano), - } -} - -func decodeRequest(id string, vals map[string]string) (Request, error) { - r := Request{ - ID: id, - Name: vals[fieldName], - Owner: vals[fieldOwner], - Visibility: reporequest.Visibility(vals[fieldVisibility]), - Description: vals[fieldDescription], - Template: vals[fieldTemplate], - Status: reporequest.Status(vals[fieldStatus]), - URL: vals[fieldURL], - Error: vals[fieldError], - RequestedBy: vals[fieldRequestedBy], - Approver: vals[fieldApprover], - RejectReason: vals[fieldRejectReason], - } - if raw := vals[fieldCreatedAt]; raw != "" { - t, err := time.Parse(time.RFC3339Nano, raw) - if err != nil { - return Request{}, fmt.Errorf("decode created_at for %q: %w", id, err) - } - r.CreatedAt = t - } - if raw := vals[fieldUpdatedAt]; raw != "" { - t, err := time.Parse(time.RFC3339Nano, raw) - if err != nil { - return Request{}, fmt.Errorf("decode updated_at for %q: %w", id, err) - } - r.UpdatedAt = t - } - return r, nil -} diff --git a/internal/reporequest/valkey/store_test.go b/internal/reporequest/valkey/store_test.go deleted file mode 100644 index 6e0035f..0000000 --- a/internal/reporequest/valkey/store_test.go +++ /dev/null @@ -1,404 +0,0 @@ -package valkey_test - -import ( - "context" - "errors" - "fmt" - "sync" - "sync/atomic" - "testing" - "time" - - "github.com/alicebob/miniredis/v2" - "github.com/stretchr/testify/assert" - "github.com/stretchr/testify/require" - - "github.com/freeCodeCamp/artemis/internal/reporequest" - "github.com/freeCodeCamp/artemis/internal/reporequest/valkey" -) - -func newStore(t *testing.T) *valkey.Store { - t.Helper() - mr := miniredis.RunT(t) - ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) - defer cancel() - s, err := valkey.New(ctx, valkey.Config{Addr: mr.Addr()}) - require.NoError(t, err) - t.Cleanup(func() { _ = s.Close() }) - - var idN int64 - s.NewID = func() string { return fmt.Sprintf("req_%03d", atomic.AddInt64(&idN, 1)) } - var clockN int64 - base := time.Date(2026, 5, 29, 12, 0, 0, 0, time.UTC) - s.Now = func() time.Time { return base.Add(time.Duration(atomic.AddInt64(&clockN, 1)) * time.Second) } - return s -} - -func sampleReq(name string) reporequest.Request { - return reporequest.Request{ - Name: name, - Owner: "freeCodeCamp-Universe", - Visibility: reporequest.VisibilityPrivate, - Description: "a repo", - RequestedBy: "octocat", - } -} - -func TestStore_CreateAndGet(t *testing.T) { - s := newStore(t) - ctx := context.Background() - - created, err := s.Create(ctx, sampleReq("my-repo")) - require.NoError(t, err) - assert.Equal(t, "req_001", created.ID) - assert.Equal(t, reporequest.StatusPending, created.Status) - - got, err := s.Get(ctx, created.ID) - require.NoError(t, err) - assert.Equal(t, "my-repo", got.Name) - assert.Equal(t, "freeCodeCamp-Universe", got.Owner) - assert.Equal(t, reporequest.VisibilityPrivate, got.Visibility) - assert.Equal(t, "octocat", got.RequestedBy) - assert.Equal(t, created.CreatedAt.UTC(), got.CreatedAt.UTC()) -} - -func TestStore_GetNotFound(t *testing.T) { - s := newStore(t) - _, err := s.Get(context.Background(), "req_missing") - assert.ErrorIs(t, err, reporequest.ErrNotFound) -} - -func TestStore_CreateDuplicateName(t *testing.T) { - s := newStore(t) - ctx := context.Background() - _, err := s.Create(ctx, sampleReq("dup")) - require.NoError(t, err) - _, err = s.Create(ctx, sampleReq("dup")) - assert.ErrorIs(t, err, reporequest.ErrAlreadyExists) -} - -func TestStore_Delete(t *testing.T) { - s := newStore(t) - ctx := context.Background() - created, err := s.Create(ctx, sampleReq("gone")) - require.NoError(t, err) - - require.NoError(t, s.Delete(ctx, created.ID)) - - _, err = s.Get(ctx, created.ID) - assert.ErrorIs(t, err, reporequest.ErrNotFound) - - _, err = s.Create(ctx, sampleReq("gone")) - assert.NoError(t, err) -} - -func TestStore_DeleteNotFound(t *testing.T) { - s := newStore(t) - assert.ErrorIs( - t, - s.Delete(context.Background(), "req_missing"), - reporequest.ErrNotFound, - ) -} - -func TestStore_DeleteResolvedRowKeepsNewerClaim(t *testing.T) { - s := newStore(t) - ctx := context.Background() - a, err := s.Create(ctx, sampleReq("x")) - require.NoError(t, err) - _, err = s.Reject(ctx, a.ID, "admin", "no") - require.NoError(t, err) - - b, err := s.Create(ctx, sampleReq("x")) - require.NoError(t, err) - require.NotEqual(t, a.ID, b.ID) - - require.NoError(t, s.Delete(ctx, a.ID)) - - _, err = s.Create(ctx, sampleReq("x")) - assert.ErrorIs(t, err, reporequest.ErrAlreadyExists) - - got, err := s.Get(ctx, b.ID) - require.NoError(t, err) - assert.Equal(t, "x", got.Name) -} - -func TestStore_MarkStaleReleasesClaimKeepsRecord(t *testing.T) { - s := newStore(t) - ctx := context.Background() - a, _ := s.Create(ctx, sampleReq("x")) - _, err := s.Approve(ctx, a.ID, "admin") - require.NoError(t, err) - _, err = s.MarkActive(ctx, a.ID, "https://github.com/freeCodeCamp-Universe/x") - require.NoError(t, err) - - stale, err := s.MarkStale(ctx, a.ID, "gone") - require.NoError(t, err) - assert.Equal(t, reporequest.StatusFailed, stale.Status) - - got, err := s.Get(ctx, a.ID) - require.NoError(t, err) - assert.Equal(t, "gone", got.Error) - - _, err = s.Create(ctx, sampleReq("x")) - assert.NoError(t, err) -} - -func TestStore_MarkStaleRequiresActive(t *testing.T) { - s := newStore(t) - a, _ := s.Create(context.Background(), sampleReq("x")) - _, err := s.MarkStale(context.Background(), a.ID, "gone") - assert.ErrorIs(t, err, reporequest.ErrNotActive) -} - -func TestStore_RejectReleasesName(t *testing.T) { - s := newStore(t) - ctx := context.Background() - r, err := s.Create(ctx, sampleReq("rel")) - require.NoError(t, err) - - rejected, err := s.Reject(ctx, r.ID, "admin1", "not needed") - require.NoError(t, err) - assert.Equal(t, reporequest.StatusRejected, rejected.Status) - assert.Equal(t, "admin1", rejected.Approver) - assert.Equal(t, "not needed", rejected.RejectReason) - - // name freed → a fresh request for the same name now succeeds. - _, err = s.Create(ctx, sampleReq("rel")) - require.NoError(t, err) -} - -func TestStore_ApproveThenActive(t *testing.T) { - s := newStore(t) - ctx := context.Background() - r, err := s.Create(ctx, sampleReq("live")) - require.NoError(t, err) - - approved, err := s.Approve(ctx, r.ID, "admin1") - require.NoError(t, err) - assert.Equal(t, reporequest.StatusApproved, approved.Status) - assert.Equal(t, "admin1", approved.Approver) - - active, err := s.MarkActive(ctx, r.ID, "https://github.com/freeCodeCamp-Universe/live") - require.NoError(t, err) - assert.Equal(t, reporequest.StatusActive, active.Status) - assert.Equal(t, "https://github.com/freeCodeCamp-Universe/live", active.URL) - - // name stays claimed after going active. - _, err = s.Create(ctx, sampleReq("live")) - assert.ErrorIs(t, err, reporequest.ErrAlreadyExists) -} - -func TestStore_MarkFailedReleasesName(t *testing.T) { - s := newStore(t) - ctx := context.Background() - r, err := s.Create(ctx, sampleReq("flaky")) - require.NoError(t, err) - _, err = s.Approve(ctx, r.ID, "admin1") - require.NoError(t, err) - - failed, err := s.MarkFailed(ctx, r.ID, "boom") - require.NoError(t, err) - assert.Equal(t, reporequest.StatusFailed, failed.Status) - assert.Equal(t, "boom", failed.Error) - - _, err = s.Create(ctx, sampleReq("flaky")) - require.NoError(t, err, "failed creation must free the name for retry") -} - -func TestStore_ApproveIsCASGuarded(t *testing.T) { - s := newStore(t) - ctx := context.Background() - r, err := s.Create(ctx, sampleReq("race")) - require.NoError(t, err) - - const racers = 8 - var wins, notPending int32 - var wg sync.WaitGroup - wg.Add(racers) - for i := 0; i < racers; i++ { - go func() { - defer wg.Done() - _, err := s.Approve(ctx, r.ID, "admin") - switch { - case err == nil: - atomic.AddInt32(&wins, 1) - case errors.Is(err, reporequest.ErrNotPending): - atomic.AddInt32(¬Pending, 1) - default: - t.Errorf("unexpected approve error: %v", err) - } - }() - } - wg.Wait() - assert.Equal(t, int32(1), atomic.LoadInt32(&wins), "exactly one admin must win the approval") - assert.Equal(t, int32(racers-1), atomic.LoadInt32(¬Pending)) -} - -func TestStore_ListSortedByCreatedAt(t *testing.T) { - s := newStore(t) - ctx := context.Background() - for _, n := range []string{"c", "a", "b"} { - _, err := s.Create(ctx, sampleReq(n)) - require.NoError(t, err) - } - list, err := s.List(ctx) - require.NoError(t, err) - require.Len(t, list, 3) - // insertion order c,a,b with strictly advancing clock → same order. - assert.Equal(t, "c", list[0].Name) - assert.Equal(t, "a", list[1].Name) - assert.Equal(t, "b", list[2].Name) -} - -func TestStore_CreateDedupeIsCaseInsensitive(t *testing.T) { - s := newStore(t) - ctx := context.Background() - _, err := s.Create(ctx, sampleReq("MyRepo")) - require.NoError(t, err) - - // GitHub repo names are case-insensitive for uniqueness — differing - // only in case must collide in the queue, not slip through to fail - // at GitHub create time. - for _, dup := range []string{"myrepo", "MYREPO", "myRepo"} { - _, err := s.Create(ctx, sampleReq(dup)) - assert.ErrorIsf(t, err, reporequest.ErrAlreadyExists, - "Create(%q) must collide with existing MyRepo", dup) - } -} - -func TestStore_RejectFreesNameCaseInsensitively(t *testing.T) { - s := newStore(t) - ctx := context.Background() - r, err := s.Create(ctx, sampleReq("MixedCase")) - require.NoError(t, err) - _, err = s.Reject(ctx, r.ID, "admin1", "") - require.NoError(t, err) - - // rejecting "MixedCase" frees the lowercased claim → a differently- - // cased resubmission succeeds. - _, err = s.Create(ctx, sampleReq("mixedcase")) - require.NoError(t, err) -} - -func TestStore_MarkActiveRequiresApproved(t *testing.T) { - tests := []struct { - name string - toStatus func(t *testing.T, s *valkey.Store, ctx context.Context, id string) - }{ - { - name: "pending", - toStatus: func(t *testing.T, s *valkey.Store, ctx context.Context, id string) {}, - }, - { - name: "active", - toStatus: func(t *testing.T, s *valkey.Store, ctx context.Context, id string) { - _, err := s.Approve(ctx, id, "admin") - require.NoError(t, err) - _, err = s.MarkActive(ctx, id, "https://github.com/freeCodeCamp-Universe/x") - require.NoError(t, err) - }, - }, - { - name: "rejected", - toStatus: func(t *testing.T, s *valkey.Store, ctx context.Context, id string) { - _, err := s.Reject(ctx, id, "admin", "no") - require.NoError(t, err) - }, - }, - { - name: "failed", - toStatus: func(t *testing.T, s *valkey.Store, ctx context.Context, id string) { - _, err := s.Approve(ctx, id, "admin") - require.NoError(t, err) - _, err = s.MarkFailed(ctx, id, "boom") - require.NoError(t, err) - }, - }, - } - for _, tc := range tests { - t.Run(tc.name, func(t *testing.T) { - s := newStore(t) - ctx := context.Background() - r, err := s.Create(ctx, sampleReq("x")) - require.NoError(t, err) - tc.toStatus(t, s, ctx, r.ID) - - _, err = s.MarkActive(ctx, r.ID, "https://github.com/freeCodeCamp-Universe/x") - assert.ErrorIs(t, err, reporequest.ErrNotPending, - "only an approved request may go active; a %s row must be guarded", tc.name) - }) - } -} - -func TestStore_MarkFailedRequiresApproved(t *testing.T) { - tests := []struct { - name string - toStatus func(t *testing.T, s *valkey.Store, ctx context.Context, id string) - }{ - { - name: "pending", - toStatus: func(t *testing.T, s *valkey.Store, ctx context.Context, id string) {}, - }, - { - name: "active", - toStatus: func(t *testing.T, s *valkey.Store, ctx context.Context, id string) { - _, err := s.Approve(ctx, id, "admin") - require.NoError(t, err) - _, err = s.MarkActive(ctx, id, "https://github.com/freeCodeCamp-Universe/x") - require.NoError(t, err) - }, - }, - { - name: "rejected", - toStatus: func(t *testing.T, s *valkey.Store, ctx context.Context, id string) { - _, err := s.Reject(ctx, id, "admin", "no") - require.NoError(t, err) - }, - }, - } - for _, tc := range tests { - t.Run(tc.name, func(t *testing.T) { - s := newStore(t) - ctx := context.Background() - r, err := s.Create(ctx, sampleReq("x")) - require.NoError(t, err) - tc.toStatus(t, s, ctx, r.ID) - - _, err = s.MarkFailed(ctx, r.ID, "boom") - assert.ErrorIs(t, err, reporequest.ErrNotPending, - "only an approved request may be marked failed; a %s row must be guarded", tc.name) - }) - } -} - -func TestStore_DeleteFailedRowKeepsReclaimedName(t *testing.T) { - s := newStore(t) - ctx := context.Background() - - a, err := s.Create(ctx, sampleReq("x")) - require.NoError(t, err) - _, err = s.Approve(ctx, a.ID, "adm") - require.NoError(t, err) - _, err = s.MarkFailed(ctx, a.ID, "boom") - require.NoError(t, err) - - b, err := s.Create(ctx, sampleReq("x")) - require.NoError(t, err) - require.NotEqual(t, a.ID, b.ID) - - require.NoError(t, s.Delete(ctx, a.ID)) - - _, err = s.Create(ctx, sampleReq("x")) - assert.ErrorIs(t, err, reporequest.ErrAlreadyExists, - "deleting a failed row must not release a name a newer pending row reclaimed") - - got, err := s.Get(ctx, b.ID) - require.NoError(t, err) - assert.Equal(t, "x", got.Name) -} - -func TestNewWithClient_NilClient(t *testing.T) { - _, err := valkey.NewWithClient(nil) - require.Error(t, err) -} From a69bd6a26bc126f7c8b1f8118d218c3b36869c10 Mon Sep 17 00:00:00 2001 From: Mrugesh Mohapatra Date: Mon, 17 Aug 2026 18:08:20 +0530 Subject: [PATCH 34/41] docs(registry): correct what happens after delete --- internal/registry/types.go | 7 +++++-- internal/registry/valkey/store.go | 5 +++-- 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/internal/registry/types.go b/internal/registry/types.go index 6e86c70..c43e335 100644 --- a/internal/registry/types.go +++ b/internal/registry/types.go @@ -60,7 +60,10 @@ type Writer interface { // Delete removes a slug from the registry (hash row + index set // member) and publishes a registry.changed event. Returns // ErrNotFound if the slug is absent. The deletion does NOT - // touch any deploy bytes in R2 — those age out via the - // post-GA cleanup cron. + // touch any deploy bytes in R2, and nothing collects them + // afterwards: registry.changed only invalidates caches, so + // gc-site never fires for the site again. The bytes and index + // rows stay until an operator runs `artemis reconcile` or the + // slug is re-registered (which resumes normal retention). Delete(ctx context.Context, slug string) error } diff --git a/internal/registry/valkey/store.go b/internal/registry/valkey/store.go index 2da9ea1..1befc4f 100644 --- a/internal/registry/valkey/store.go +++ b/internal/registry/valkey/store.go @@ -358,8 +358,9 @@ func (s *Store) UpdateTeams(ctx context.Context, slug string, teams []string) (S // Delete removes the slug's hash row + index-set member and // publishes a registry.changed event. Returns ErrNotFound if the -// slug is absent. R2 deploy bytes are NOT touched — those age out -// via the post-GA cleanup cron. +// slug is absent. R2 deploy bytes are NOT touched, and no job +// collects them afterwards — see registry.Writer.Delete for the +// retention consequences. func (s *Store) Delete(ctx context.Context, slug string) error { if slug == "" { return errors.New("registry: empty slug") From e6361f3f066be2590febd593b00df495ec47a4dc Mon Sep 17 00:00:00 2001 From: Mrugesh Mohapatra Date: Mon, 17 Aug 2026 18:08:45 +0530 Subject: [PATCH 35/41] docs: date the design-doc claims to what shipped --- docs/design/0001-durable-execution-model.md | 4 ++-- docs/design/0004-drift-detection-and-alerting.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/design/0001-durable-execution-model.md b/docs/design/0001-durable-execution-model.md index ed1d43d..45e994e 100644 --- a/docs/design/0001-durable-execution-model.md +++ b/docs/design/0001-durable-execution-model.md @@ -170,8 +170,8 @@ Audited this plan against the Universe Architecture ADRs. Building blocks + plac Two latent gaps confirmed in code against the shipped v1.3.0 tag. Canonical record + full citations: Universe ADR-020 § "2026-07-05 — known limitations (v1.3.0)" (`~/DEV/fCC-U/Architecture/decisions/020-durable-execution.md`). -- **Reconcile drift-audit backstop (E4) not wired** — `WorkflowReconcile` consumes `site.reconcile` (`cmd/artemis/gcworkflows.go:79-94`) but no producer emits it; the §5 E4 backstop has never run in prod. -- **Outbox relay duplicate-publish** — `FetchUnpublished` (`internal/pg/outbox.go:41`) lacks `FOR UPDATE SKIP LOCKED`; `runRelayLoop` runs per-replica (`cmd/artemis/main.go:262`, `replicaCount: 3`), so duplicate publishes are possible (bounded by concurrency-key + idempotency). +- **Reconcile drift-audit backstop (E4) not wired** — at v1.3.0 a `WorkflowReconcile` consumer existed with no producer; the §5 E4 backstop had never run in prod. Both sides are gone now: the producer was retired (next block) and the consumer plus its `WorkflowReconcile` constant were deleted in the drift-at-source wave. +- **Outbox relay duplicate-publish** — at v1.3.0 the relay read via a plain `FetchUnpublished` with no `FOR UPDATE SKIP LOCKED` while running per-replica, so duplicate publishes were possible. Fixed since: the relay claims batches with `FOR UPDATE SKIP LOCKED` and a 5-minute claim TTL (`internal/pg/outbox.go` claimBatch), and `FetchUnpublished` itself is deleted. ### Update (2026-08-16) diff --git a/docs/design/0004-drift-detection-and-alerting.md b/docs/design/0004-drift-detection-and-alerting.md index 249c7c9..10d36da 100644 --- a/docs/design/0004-drift-detection-and-alerting.md +++ b/docs/design/0004-drift-detection-and-alerting.md @@ -8,7 +8,7 @@ The reconciler does not repair on a schedule. A read-only sweep finds drift and ## Why -The reconcile cron ran every day at 04:00 UTC and repaired nothing. Postgres holds zero `gc.reconcile` audit rows, against 39 `gc.tombstone` rows from the retention GC. The cause was a keyspace error: the scheduler listed registry slugs (`test`), but the bytes are under storage dirnames (`test.freecode.camp`). Every sweep looked at a prefix that does not exist. +The reconcile cron ran every day at 04:00 UTC and repaired nothing. At retirement (2026-08-16) Postgres held zero `gc.reconcile` audit rows against 39 `gc.tombstone` rows from the retention GC; the first `gc.reconcile` rows (2, on 2026-08-17) came from the manual `artemis reconcile --apply` run that validated the human-run repair path. The cause was a keyspace error: the scheduler listed registry slugs (`test`), but the bytes are under storage dirnames (`test.freecode.camp`). Every sweep looked at a prefix that does not exist. A read-only sweep of production on 2026-08-16 measured the real drift: From ce4bc43d4c41d7b916ff0aa681ce07a0a0be1d7d Mon Sep 17 00:00:00 2001 From: Mrugesh Mohapatra Date: Mon, 17 Aug 2026 18:09:32 +0530 Subject: [PATCH 36/41] docs: bring the architecture up to the shipped code --- docs/ARCHITECTURE.md | 50 +++++++++++++++++++++++--------------------- 1 file changed, 26 insertions(+), 24 deletions(-) diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 5d1892a..865e80a 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -62,7 +62,7 @@ A deploy has three calls. Artemis makes the deploy id, and the client never choo 1. Artemis makes the deploy id. The shape is `-`, in UTC. 1. Artemis signs a **deploy-session JWT**. The token holds the login, the site, and the deploy id. The default life is 15 minutes. -Init writes nothing to R2, and the deploy prefix does not exist yet. Init does write one `audit_log` row in Postgres. That write is best effort, and a failure does not fail the request. +Init writes nothing to R2, and the deploy prefix does not exist yet. Init does write to Postgres, twice: one `audit_log` row, and one `deploys` row with the state `pending`. The pending row is what lets the cleanup job collect a deploy that uploads bytes and never finalizes — every read path filters on the state `active`, so the pending row is invisible until finalize promotes it. Both writes are best effort, and a failure does not fail the request. ### Step 2 — upload @@ -82,14 +82,17 @@ There is no staging area. Each object lands at its final key immediately. The de Artemis runs these gates in order, and it stops at the first failure: -| Order | Gate | Failure | -| ----- | ---------------------------------------------- | -------------------------- | -| 1 | The mode must be `preview` or `production` | 400 | -| 2 | The file manifest must not be empty | 400 | -| 3 | The manifest must hold a root `index.html` | 422 | -| 4 | R2 must hold every file in the manifest | 422, with the missing list | -| 5 | Artemis writes the `_artemis_meta.json` marker | 502 | -| 6 | Artemis measures the deploy size | not fatal | +| Order | Gate | Failure | +| ----- | ----------------------------------------------- | -------------------------- | +| 1 | The JWT claims must be present on the request | 401 | +| 2 | The URL deploy id must equal the JWT deploy id | 403 | +| 3 | The mode must be `preview` or `production` | 400 | +| 4 | The file manifest must not be empty | 400 | +| 5 | The manifest must hold a root `index.html` | 422 | +| 6 | R2 must hold every file in the manifest | 422, with the missing list | +| 7 | Artemis writes the `_artemis_meta.json` marker | 502 | + +Artemis then measures the deploy size. That step is not a gate: a failed measurement records zero bytes and the finalize continues. Artemis then takes a **per-site lock** in Postgres, and it does the last steps inside that lock: @@ -155,7 +158,7 @@ The move is a copy and then a delete, for each object. It has no rollback. A fai `DELETE /api/site/{slug}` has two behaviours: - **Without `?purge=true`** — artemis removes the registry row only. It touches no R2 bytes. It returns 204. -- **With `?purge=true`** — artemis moves the full `/` prefix into `_trash//`, writes a whole-site tombstone, and then removes the registry row. It returns 200. +- **With `?purge=true`** — artemis writes a whole-site tombstone, moves the full `/` prefix into `_trash//`, and then removes the registry row. It returns 200. The prefix is the storage dirname (section 9), not the slug, and the tombstone row lands before the move — the order every removal in artemis follows. The purge moves the alias objects too, because they are under the same `/` prefix. @@ -213,7 +216,7 @@ Postgres refuses a drift 1 or a drift 3 repair for a deploy that holds a tombsto The reconciler stops the run for a site if Postgres holds deploys for that site and the R2 listing returns no deploy prefix at all. An empty listing looks the same as total drift, and total drift would delete every index row for the site. The reconciler treats it as a fault instead, and it reports the fault. This is the shape the name defect in section 10 had, and it is also the shape of a wrong bucket or a wrong prefix. -The blast cap limits how many deploys one run can move to the trash or remove from the index. A run that plans more repairs than the cap repairs the oldest deploys first, reports the cap, and leaves the remainder for the next run. The cap does not limit the repairs that write an index row, because those repairs remove nothing. +The blast cap limits how many deploys one run can move to the trash or remove from the index. A run that plans more repairs than the cap repairs the oldest deploys first, reports the cap, and leaves the remainder for the next run. A cap of zero refuses every destructive repair — it is a refusal, not an absence of limit. The cap does not limit the repairs that write an index row, because those repairs remove nothing, and it does not shorten a dry-run report: the report always names every drifted deploy and warns separately that a live run would be capped. One selection function owns "the oldest N" for every capped path, so the retention job, the reconciler and the purge cannot disagree about which deploys survive a ceiling. Two limits are important: @@ -260,7 +263,16 @@ Condition 5 is important. The retention window applies only to a deploy that hol Before it moves a deploy, `gc-site` takes the per-site lock and reads the **live R2 aliases** again. The plan uses the Postgres alias rows, but the execution uses the R2 objects. A deploy that became live in the interval is skipped. -The `gc-site` workflow deletes no bytes. It only moves a prefix into the trash. The `drift-detect` cron writes nothing at all: its store and its mover are read-only types, and every write method returns an error. The `tombstone-purge` workflow is the only hard delete in the service. +The `gc-site` workflow deletes no bytes. It only moves a prefix into the trash. The `drift-detect` cron writes nothing at all: its store and its mover are read-only types, and every write method returns an error. The `tombstone-purge` workflow is the only hard delete in the service. It is bounded by the same blast cap as the other destructive paths, it deletes the most overdue trash first, and one site's failure defers only that site — the rest of the run continues and the workflow still reports red. + +### The write-ordering rule + +Every removal writes its two side effects in a fixed order, and the two orders are opposites for a reason. + +- **Moving bytes to the trash** (`gc-site`, the reconciler, deploy delete, site purge): the tombstone row first, the byte move second. Bytes in `_trash/` with no tombstone are invisible to every job forever — the purge walks the `tombstones` table, and the drift sweep lists the site prefix, not the trash. The benign failure is the inverse: a row whose bytes never moved surfaces as drift, and clears when the purge drops the row after the recovery window. +- **Hard-deleting the trash** (`tombstone-purge`): the byte delete first, the row clear second. A surviving row keeps the idempotent delete retryable on the next run; clearing the row first would drop the only record that bytes remain. + +Each background job also carries an explicit execution budget, so the engine default never decides when a half-finished run is killed. ## 8. How identity and authorization work @@ -322,6 +334,8 @@ The default `DEPLOY_PREFIX_FORMAT` is `/deploys/-/`. With this fo R2 keys always come from the raw slug, rendered through the template. The `deploys` table, the `aliases` table, the `tombstones` table, and the `site.changed` outbox payload always hold the storage dirname, never the slug. Each write path converts the slug to a dirname before it touches one of these stores. +The `audit_log` table is the one exception, and it holds **both** names: the HTTP handlers write the slug (via the request-scoped telemetry), while the GC auditors write the dirname the sweep enumerates. No single query over `audit_log.site` returns a site's complete history under a format where the two names differ. The table is append-only at the database level (triggers reject UPDATE, DELETE and TRUNCATE), so the split cannot be repaired by rewriting history — only by converging the writers and dating the cutover. + A caller that reads the registry and skips this conversion sends the bare slug to a store that expects the dirname. The query or the R2 prefix then matches nothing, even though the site is real. Section 10 describes one case where this happened. ### The registry, in detail @@ -341,15 +355,3 @@ Team revocation is therefore eventually consistent, and not immediate. 1. **One Postgres advisory lock, keyed by the site, serializes each mutation of that site.** Finalize, promote, rollback, delete, restore, purge, the cleanup job, and each reconciler repair all take the same key. The timeout is 30 seconds, and a contended request gets 409. The second invariant has one dangerous limit. With no Postgres, the lock becomes a silent no-op, and concurrent alias writes race with no error. - -## 10. Divergence between this code and the deployed release - -This document describes what the code at HEAD does. The deployed release still runs a nightly repair cron. HEAD removes it. - -The deployed `reconcile-scheduler` workflow reads the registry slugs and publishes each one straight into a `site.reconcile` event, with no conversion to a storage dirname (section 9 defines both names). Every other write path converts the slug first. The deployed reconciler therefore builds an R2 prefix and a Postgres query from a name that no store recognizes. With the default `DEPLOY_PREFIX_FORMAT`, the slug and the dirname are the same string, so the reconciler still works. With a format that adds a suffix, such as the deployed `.freecode.camp/deploys/-/`, the slug and the dirname differ, and the reconciler finds nothing on either side. It reports zero drift and completes without error, on every site, every night. This is why the split must stay documented even though the default format hides it. - -A read-only sweep at HEAD replaces that scheduler. The sweep converts each registry slug to its storage dirname, and it also reads the site names that only the index knows. It writes nothing. - -The deployed reconciler also carries a second hazard. It reads the two sides one time and then repairs from that first read, with no lock and no second read. It has no blast cap, and it removes an index row of any age. The retirement removes the hazard, because no scheduled job repairs anything. The repair path at HEAD takes the site lock and reads the state again inside that lock, and only an operator can start it. - -Until the release carrying this change is deployed, the running service still shows the old behaviour: the nightly reconcile workflow completes and repairs nothing. From 8eabe1078253193a85998443a318969e27b0432a Mon Sep 17 00:00:00 2001 From: Mrugesh Mohapatra Date: Mon, 17 Aug 2026 18:09:48 +0530 Subject: [PATCH 37/41] docs: drop the reading step for the deleted section --- docs/ORIENTATION.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/ORIENTATION.md b/docs/ORIENTATION.md index 6034844..24681ef 100644 --- a/docs/ORIENTATION.md +++ b/docs/ORIENTATION.md @@ -7,7 +7,6 @@ This page is for a new contributor. It gives the read sequence for the artemis a 1. Read the root [`README.md`](../README.md). It tells you what artemis is and how to start the service locally. 1. Read [`ARCHITECTURE.md`](ARCHITECTURE.md), sections 1 to 3. You learn the problem artemis solves, the parts it depends on, and the deploy lifecycle. 1. Read [`ARCHITECTURE.md`](ARCHITECTURE.md), sections 4 to 9. You learn the alias model, removal and recovery, reconciliation, background work, authorization, and where each piece of state lives. -1. Read section 10 of the same document. It lists the known divergence between the code and the deployed release. 1. Read ADR-016 in the Universe platform repo (`Architecture/decisions/016-deploy-proxy.md`). It is the authoritative specification for the API surface and the per-site authorization model. 1. Read [`design/0001-durable-execution-model.md`](design/0001-durable-execution-model.md). You learn why Postgres and Hatchet are part of the design, and the safety invariants of the retention GC. 1. Use [`README.md`](README.md) in this directory as a reference. Look up routes, configuration variables, observability, and the test suites there. Do not read it end to end. From eb8f44bb61a6a12be596b69396fedb93e4aa85ea Mon Sep 17 00:00:00 2001 From: Mrugesh Mohapatra Date: Mon, 17 Aug 2026 18:12:31 +0530 Subject: [PATCH 38/41] chore: drop orphaned test fixtures --- cmd/artemis/driftfixtures_test.go | 63 ------------------------------ cmd/artemis/workflowerrors_test.go | 6 --- 2 files changed, 69 deletions(-) diff --git a/cmd/artemis/driftfixtures_test.go b/cmd/artemis/driftfixtures_test.go index dc0e373..980aaf8 100644 --- a/cmd/artemis/driftfixtures_test.go +++ b/cmd/artemis/driftfixtures_test.go @@ -2,71 +2,8 @@ package main import ( "context" - "errors" - "log/slog" - "sync" - "time" - - "github.com/freeCodeCamp/artemis/internal/gc" ) -type msgCapture struct { - mu sync.Mutex - msgs []string -} - -func (h *msgCapture) Enabled(context.Context, slog.Level) bool { return true } -func (h *msgCapture) Handle(_ context.Context, r slog.Record) error { - h.mu.Lock() - defer h.mu.Unlock() - h.msgs = append(h.msgs, r.Message) - return nil -} -func (h *msgCapture) WithAttrs([]slog.Attr) slog.Handler { return h } -func (h *msgCapture) WithGroup(string) slog.Handler { return h } - -func (h *msgCapture) saw(msg string) bool { - h.mu.Lock() - defer h.mu.Unlock() - for _, m := range h.msgs { - if m == msg { - return true - } - } - return false -} - -type driftingStore struct{ nopReconcileStore } - -func (driftingStore) DeploysForSite(context.Context, string) ([]gc.Deploy, error) { - return []gc.Deploy{{ID: "ghost", Mtime: time.Now().Add(-30 * 24 * time.Hour)}}, nil -} - -func (driftingStore) AliasTargets(context.Context, string) (map[string]struct{}, time.Time, error) { - return map[string]struct{}{"ghost": {}}, time.Time{}, nil -} - -func (driftingStore) RecordTombstone(context.Context, string, string, int64) error { - return errors.New("pg down") -} - type staticLister struct{ keys []string } func (l staticLister) ListPrefix(context.Context, string) ([]string, error) { return l.keys, nil } - -type passthroughSession struct{} - -func (passthroughSession) WithSiteLock(_ context.Context, _ string, fn func() error) error { - return fn() -} -func (passthroughSession) Close(context.Context) {} - -type passthroughLocker struct{} - -func (passthroughLocker) NewLockSession(context.Context) (gc.LockSession, error) { - return passthroughSession{}, nil -} - -type nopMover struct{} - -func (nopMover) MovePrefix(context.Context, string, string) (int, error) { return 0, nil } diff --git a/cmd/artemis/workflowerrors_test.go b/cmd/artemis/workflowerrors_test.go index e4b9317..4d4d65e 100644 --- a/cmd/artemis/workflowerrors_test.go +++ b/cmd/artemis/workflowerrors_test.go @@ -14,12 +14,6 @@ import ( "github.com/freeCodeCamp/artemis/internal/worker" ) -type failingLister struct{ err error } - -func (l *failingLister) ListPrefix(context.Context, string) ([]string, error) { - return nil, l.err -} - func defByName(t *testing.T, defs []worker.WorkflowDef, name string) worker.WorkflowDef { t.Helper() for _, d := range defs { From 6b1ad31a7f68e5ecd241950a8f936f057dd10696 Mon Sep 17 00:00:00 2001 From: Mrugesh Mohapatra Date: Mon, 17 Aug 2026 18:35:43 +0530 Subject: [PATCH 39/41] fix(auth): classify 429s and close review findings --- .env.example | 2 +- cmd/artemis/driftalert_threshold_test.go | 12 +++++ docs/ARCHITECTURE.md | 4 +- internal/auth/github.go | 9 ++-- internal/auth/github_cachebound_test.go | 24 +++++++--- .../auth/github_secondary_ratelimit_test.go | 45 +++++++++++++++++++ internal/observability/sentry.go | 2 + 7 files changed, 85 insertions(+), 13 deletions(-) diff --git a/.env.example b/.env.example index 554490a..7cba4ea 100644 --- a/.env.example +++ b/.env.example @@ -65,7 +65,7 @@ VALKEY_ADDR=localhost:6379 # CLEANUP_RETENTION_DAYS=7 # days before a superseded deploy is GC-eligible # CLEANUP_RECENT_KEEP=3 # newest N deploys per site always kept # CLEANUP_GRACE=72h # min deploy age before GC; must be >= JWT_TTL_SECONDS -# CLEANUP_BLAST_CAP=10 # max deletes per sweep, oldest first; 0 refuses every repair +# CLEANUP_BLAST_CAP=10 # max deletes per sweep, oldest first; 0 refuses every destructive repair # CLEANUP_TRASH_PREFIX=_trash/ # R2 prefix for tombstoned objects # CLEANUP_RECOVERY_DAYS=7 # days a tombstone survives before hard purge # CLEANUP_DRY_RUN= # 1/true: plan-only, execute nothing diff --git a/cmd/artemis/driftalert_threshold_test.go b/cmd/artemis/driftalert_threshold_test.go index b67b28f..d528ebb 100644 --- a/cmd/artemis/driftalert_threshold_test.go +++ b/cmd/artemis/driftalert_threshold_test.go @@ -3,6 +3,7 @@ package main import ( "testing" + "github.com/freeCodeCamp/artemis/internal/observability" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" ) @@ -56,3 +57,14 @@ func TestClassifyDrift_AliasedMissingOutranksTheReclaimableThreshold(t *testing. assert.Equal(t, opDriftAliasedMissing, v.Op, "a live site serving nothing must not be masked by a large but harmless reclaimable count") } + +func TestEveryDriftVerdictOpIsCronShaped(t *testing.T) { + t.Parallel() + + for _, op := range []string{opDriftSweep, opDriftSelfCheck, opDriftUnreadable, opDriftAliasedMissing, opDriftReclaimable} { + assert.True(t, observability.IsCronShaped(op), + "op %s bypasses the transient-rate tracker only if cronShapedOps lists it; this test lives "+ + "beside the op constants so adding a sixth verdict here fails until the map learns it — "+ + "the observability-side test could only restate the map against itself", op) + } +} diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 865e80a..e936598 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -62,7 +62,7 @@ A deploy has three calls. Artemis makes the deploy id, and the client never choo 1. Artemis makes the deploy id. The shape is `-`, in UTC. 1. Artemis signs a **deploy-session JWT**. The token holds the login, the site, and the deploy id. The default life is 15 minutes. -Init writes nothing to R2, and the deploy prefix does not exist yet. Init does write to Postgres, twice: one `audit_log` row, and one `deploys` row with the state `pending`. The pending row is what lets the cleanup job collect a deploy that uploads bytes and never finalizes — every read path filters on the state `active`, so the pending row is invisible until finalize promotes it. Both writes are best effort, and a failure does not fail the request. +Init writes nothing to R2, and the deploy prefix does not exist yet. Init does write to Postgres, twice: one `audit_log` row, and one `deploys` row with the state `pending`. The pending row is what lets the cleanup job collect a deploy that uploads bytes and never finalizes — every read that lists deploys filters on the state `active`, so the pending row is invisible to retention planning, the drift denominator and the API until finalize promotes it. Its only readers are the expiry query the cleanup job runs, and the site enumerator, which does not filter by state. Both writes are best effort, and a failure does not fail the request. ### Step 2 — upload @@ -269,7 +269,7 @@ The `gc-site` workflow deletes no bytes. It only moves a prefix into the trash. Every removal writes its two side effects in a fixed order, and the two orders are opposites for a reason. -- **Moving bytes to the trash** (`gc-site`, the reconciler, deploy delete, site purge): the tombstone row first, the byte move second. Bytes in `_trash/` with no tombstone are invisible to every job forever — the purge walks the `tombstones` table, and the drift sweep lists the site prefix, not the trash. The benign failure is the inverse: a row whose bytes never moved surfaces as drift, and clears when the purge drops the row after the recovery window. +- **Moving bytes to the trash** (`gc-site`, the reconciler, deploy delete, site purge): the tombstone row first, the byte move second. Bytes in `_trash/` with no tombstone are invisible to every job forever — the purge walks the `tombstones` table, and the drift sweep lists the site prefix, not the trash. The benign failure is the inverse: a row whose bytes never moved leaves the deploy out of the index while its bytes sit at the live prefix. The nightly sweep reports it, the purge eventually drops the tombstone (which is what un-blocks reindexing), and an operator `artemis reconcile` restores the row — the bytes are visible and recoverable at every step, never silently gone. - **Hard-deleting the trash** (`tombstone-purge`): the byte delete first, the row clear second. A surviving row keeps the idempotent delete retryable on the next run; clearing the row first would drop the only record that bytes remain. Each background job also carries an explicit execution budget, so the engine default never decides when a half-finished run is killed. diff --git a/internal/auth/github.go b/internal/auth/github.go index 7a5ec0d..10ede28 100644 --- a/internal/auth/github.go +++ b/internal/auth/github.go @@ -170,7 +170,7 @@ func (c *GitHubClient) fetchUser(ctx context.Context, cacheKey, token string) (s case resp.StatusCode == http.StatusUnauthorized: c.cacheNegative(cacheKey, ErrGitHubUnauthenticated) return "", ErrGitHubUnauthenticated - case resp.StatusCode == http.StatusForbidden && isRateLimited(resp): + case isRateLimited(resp): // transient — DO NOT cache. return "", ErrGitHubRateLimited case resp.StatusCode == http.StatusForbidden: @@ -335,7 +335,7 @@ func (c *GitHubClient) fetchTeamMembership(ctx context.Context, token, user, tea member = m.State == "active" case resp.StatusCode == http.StatusNotFound: member = false - case resp.StatusCode == http.StatusForbidden && isRateLimited(resp): + case isRateLimited(resp): return false, ErrGitHubRateLimited case resp.StatusCode >= 500: return false, ErrGitHubUnavailable @@ -445,7 +445,7 @@ func (c *GitHubClient) fetchUserTeams(ctx context.Context, cacheKey, token strin // fall through case resp.StatusCode == http.StatusUnauthorized: return nil, ErrGitHubUnauthenticated - case resp.StatusCode == http.StatusForbidden && isRateLimited(resp): + case isRateLimited(resp): return nil, ErrGitHubRateLimited case resp.StatusCode == http.StatusForbidden: return nil, ErrGitHubUnauthenticated @@ -529,6 +529,9 @@ func IsGitHubUnauthenticated(err error) bool { return errors.Is(err, ErrGitHubUn // this check they fell through to the plain-403 branch and were // negative-cached as unauthenticated for up to negCacheCap. func isRateLimited(resp *http.Response) bool { + if resp.StatusCode == http.StatusTooManyRequests { + return true + } if resp.Header.Get("X-RateLimit-Remaining") == "0" { return true } diff --git a/internal/auth/github_cachebound_test.go b/internal/auth/github_cachebound_test.go index e04e52d..da4d2b1 100644 --- a/internal/auth/github_cachebound_test.go +++ b/internal/auth/github_cachebound_test.go @@ -1,11 +1,15 @@ package auth import ( + "context" "fmt" + "net/http" + "net/http/httptest" "testing" "time" "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" ) func TestGitHubClient_CachesAreBoundedAcrossTokenChurn(t *testing.T) { @@ -20,14 +24,20 @@ func TestGitHubClient_CachesAreBoundedAcrossTokenChurn(t *testing.T) { "every distinct bearer ever presented used to leave a permanent map entry; CI tokens rotate and "+ "pods live for weeks, so the maps grew monotonically for the life of the process") - c.mu.Lock() - for i := 0; i < maxCacheEntries*2; i++ { - key := teamCacheKey{user: fmt.Sprintf("u-%d", i), team: "t"} - pruneMap(c.teamCache, func(e teamCacheEntry) bool { return !e.expires.After(c.now()) }, maxCacheEntries) - c.teamCache[key] = teamCacheEntry{member: true, expires: clock.Add(time.Minute)} + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(http.StatusOK) + _, _ = w.Write([]byte(`{"state":"active"}`)) + })) + t.Cleanup(srv.Close) + tc := NewGitHubClient(GitHubClientConfig{APIBase: srv.URL, Org: "freeCodeCamp", + Now: func() time.Time { return clock }}) + for i := 0; i < maxCacheEntries+64; i++ { + _, err := tc.IsTeamMember(context.Background(), "tok", fmt.Sprintf("u-%d", i), "t") + require.NoError(t, err) } - c.mu.Unlock() - assert.LessOrEqual(t, len(c.teamCache), maxCacheEntries+1) + assert.LessOrEqual(t, len(tc.teamCache), maxCacheEntries, + "driven through IsTeamMember so the assertion fails if fetchTeamMembership stops pruning; the "+ + "first version of this test called pruneMap from its own loop and measured its own bookkeeping") } func TestPruneMap_DropsExpiredBeforeLive(t *testing.T) { diff --git a/internal/auth/github_secondary_ratelimit_test.go b/internal/auth/github_secondary_ratelimit_test.go index 1ac71c1..d7b8767 100644 --- a/internal/auth/github_secondary_ratelimit_test.go +++ b/internal/auth/github_secondary_ratelimit_test.go @@ -36,6 +36,7 @@ func TestValidateToken_SecondaryRateLimitIsNotAnAuthFailure(t *testing.T) { "a secondary-limit 403 carries Retry-After with non-zero X-RateLimit-Remaining; classifying it as "+ "unauthenticated tells the operator a working credential is bad") assert.False(t, IsGitHubUnauthenticated(err)) + assert.EqualValues(t, 1, calls.Load(), "one upstream probe, no retry loop inside the client") } func TestValidateToken_SecondaryRateLimitIsNeverNegativeCached(t *testing.T) { @@ -60,6 +61,50 @@ func TestIsTeamMember_SecondaryRateLimitSurfacesAsRateLimited(t *testing.T) { _, err := c.IsTeamMember(context.Background(), "tok", "alice", "team-eng") + require.Error(t, err) + assert.True(t, IsGitHubRateLimited(err)) + assert.EqualValues(t, 1, calls.Load(), "one upstream probe, no retry loop inside the client") +} + +func TestValidateToken_429IsRateLimitedNotUnauthenticated(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Retry-After", "30") + w.WriteHeader(http.StatusTooManyRequests) + })) + t.Cleanup(srv.Close) + c := NewGitHubClient(GitHubClientConfig{APIBase: srv.URL}) + + _, err := c.ValidateToken(context.Background(), "tok-429") + + require.Error(t, err) + assert.True(t, IsGitHubRateLimited(err), + "GitHub documents 403 OR 429 for both limit classes; a 429 fell through to the default branch "+ + "and reached the operator as 401 invalid-token") + assert.False(t, IsGitHubUnauthenticated(err)) +} + +func TestIsTeamMember_429IsRateLimited(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(http.StatusTooManyRequests) + })) + t.Cleanup(srv.Close) + c := NewGitHubClient(GitHubClientConfig{APIBase: srv.URL, Org: "freeCodeCamp"}) + + _, err := c.IsTeamMember(context.Background(), "tok", "alice", "team-eng") + + require.Error(t, err) + assert.True(t, IsGitHubRateLimited(err)) +} + +func TestUserTeams_429IsRateLimited(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(http.StatusTooManyRequests) + })) + t.Cleanup(srv.Close) + c := NewGitHubClient(GitHubClientConfig{APIBase: srv.URL}) + + _, err := c.UserTeams(context.Background(), "tok") + require.Error(t, err) assert.True(t, IsGitHubRateLimited(err)) } diff --git a/internal/observability/sentry.go b/internal/observability/sentry.go index 228804d..5b1912d 100644 --- a/internal/observability/sentry.go +++ b/internal/observability/sentry.go @@ -374,6 +374,8 @@ func NewSlogHandler(minLevel slog.Level) slog.Handler { }.NewSentryHandler(context.Background()) } +func IsCronShaped(op string) bool { return cronShapedOps[op] } + var cronShapedOps = map[string]bool{ "drift.sweep": true, "drift.selfcheck": true, From 51848adcfe2a8fb914456619838a0589efb8e92a Mon Sep 17 00:00:00 2001 From: Mrugesh Mohapatra Date: Mon, 17 Aug 2026 18:36:10 +0530 Subject: [PATCH 40/41] docs: correct the reaping-order recovery claims --- docs/ONBOARDING.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/ONBOARDING.md b/docs/ONBOARDING.md index fc9394e..f2628fa 100644 --- a/docs/ONBOARDING.md +++ b/docs/ONBOARDING.md @@ -154,9 +154,9 @@ Safety rails on the repair path: the site lock, a **second read inside the lock* ### 7.4 Why both reaping paths write the row first -Every reaping path records the tombstone row **before** it moves the bytes — `gc-site` at `internal/gc/gcsite.go:135` then `:138`, `reconcile` at `internal/gc/reconcile.go:286` then `:290`. +Every reaping path records the tombstone row **before** it moves the bytes — `gc-site` at `internal/gc/gcsite.go:135` then `:138`, `reconcile` at `internal/gc/reconcile.go:284` then `:288`. -That order is forced by the purge being row-driven. Bytes moved into `_trash/` without a tombstone are invisible to `tombstone-purge` (which walks `tombstones`), invisible to the index, and invisible to `reconcile` (which lists the *site* prefix, not `_trash/`) — a permanent, undetectable leak. The inverse failure is bounded: a row with its bytes still at the deploy prefix shows up as reindex drift, which the nightly sweep reports. `reconcile` cannot repair it immediately — `ReindexDeploy` refuses while a tombstone for that id stands (`internal/pg/repo.go:46`) — so the bytes clear once `tombstone-purge` drops the row after `CLEANUP_RECOVERY_DAYS`. Both paths log `tombstone_move_deferred` when they land in that state. +That order is forced by the purge being row-driven. Bytes moved into `_trash/` without a tombstone are invisible to `tombstone-purge` (which walks `tombstones`), invisible to the index, and invisible to `reconcile` (which lists the *site* prefix, not `_trash/`) — a permanent, undetectable leak. The inverse failure is bounded and visible: a row with its bytes still at the deploy prefix shows up as reindex drift, which the nightly sweep reports. `ReindexDeploy` refuses while the tombstone stands (`internal/pg/repo.go:46`); once `tombstone-purge` drops the row after `CLEANUP_RECOVERY_DAYS`, an operator `artemis reconcile` re-indexes the bytes. The purge clears the row, not the bytes — reclaiming or restoring them is the reconcile's job. Both paths log `tombstone_move_deferred` when they land in that state. `gc-site` carried the leaky order until the drift-at-source sprint; if you find a doc or comment claiming otherwise, it predates that change. @@ -205,7 +205,7 @@ Verified traps, each a real line of code. None of these are hypothetical. 1. **There is no "already finalized" guard on upload.** A valid JWT can keep writing into a prefix that is already the live production target, for the rest of its TTL. Known and accepted; see design 0005. 1. **A deploy row exists from `init`, not from `finalize`.** `deploy.init` writes `state = 'pending'` (`internal/pg/pending.go`); `FinalizeAtomic`'s existing `ON CONFLICT ... SET state = 'active'` promotes it with no extra write. Every read filters `state = 'active'`, so a pending row is invisible to retention planning, the drift denominator and the API — its only reader is `ExpiredPendingDeploys`, which `gc-site` uses to reap sessions abandoned past the grace window. The write is best-effort: a failure logs and raises to Sentry but never fails the deploy. 1. **`site-purge` writes a sentinel tombstone with `id = ''`**, and that row now blocks reindexing of *every* deploy in that site until the recovery window clears it. That is deliberate, added this week, and easy to mistake for a bug. -1. **Dead code that looks live.** `worker.RegisterDeployWorkflows` is never called outside tests — finalize, promote and rollback all run inline in the HTTP handlers. +1. **Finalize, promote and rollback never touch the workflow engine.** All three run inline in the HTTP handlers; the only registered workflows are the three GC jobs. A `RegisterDeployWorkflows` shim once suggested otherwise and has been deleted. ______________________________________________________________________ From 96919c43e02d6ddfcdedfed845d1bdb3bb37a7e6 Mon Sep 17 00:00:00 2001 From: Mrugesh Mohapatra Date: Mon, 17 Aug 2026 18:44:23 +0530 Subject: [PATCH 41/41] docs: record wave outcome and open decisions --- docs/design/0005-drift-at-source.md | 66 ++++++++++++++++++++--------- 1 file changed, 46 insertions(+), 20 deletions(-) diff --git a/docs/design/0005-drift-at-source.md b/docs/design/0005-drift-at-source.md index 0b53b98..1a79b2b 100644 --- a/docs/design/0005-drift-at-source.md +++ b/docs/design/0005-drift-at-source.md @@ -181,26 +181,26 @@ ______________________________________________________________________ Everything verified during this audit, with a decision against each. "Accept, documented" is a real disposition; silence is not. -| # | Finding | Disposition | -| --- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| 1 | `LiveAliases` receives a dirname, alias format expects a slug → always 404, safety net inert in prod | **P0-2**, fix now | -| 2 | Whole gc/CLI suite runs under a format where slug == dirname, hiding the entire bug class | **P0-1**, fix now | -| 3 | `CLEANUP_BLAST_CAP` has no default → `0` → unlimited destruction | **P0-3**, fix now | -| 4 | `gc-site` moves bytes before writing the tombstone row | **P0-4**, fix now | -| 5 | `reconcile` records tombstones with hardcoded `bytes = 0` | **P0-4**, in passing | -| 6 | Abandoned `deploy.init` sessions leave unowned bytes (~18/month) | **P2**, the root cause | -| 7 | Two keyspaces with no type separation | **P1** | -| 8 | Postgres stores dirnames, registry stores slugs | **Out of scope** — migration; P1's types make it survivable | -| 9 | `runDriftReport` is called with no argv (`cmd/artemis/main.go:49`) so `driftreport ` silently ignores it; and any unrecognised subcommand falls through to `run()` (`:62`), i.e. **a mistyped subcommand starts the server** | **P0-adjacent** — operators started running these subcommands against production *this week*. A typo that boots a server, and a report that ignores the argument an operator typed, are both how a run gets misread as authoritative. Fix with P0: reject unknown subcommands, reject unexpected args. | -| 10 | A finalized deploy remains writable for the JWT's remaining TTL (up to 15 min) | **Accept, documented.** Real, but requires an authorized token holder; tightening it means invalidating the JWT at finalize, which is its own design. Record in ONBOARDING traps. | -| 11 | `outbox` has no retention — unbounded growth | **Backlog.** Small table, slow growth, no correctness impact. Needs a purge job eventually; not part of this wave. | -| 12 | Dead worker code paths | **Backlog**, cosmetic. | -| 13 | `RequireScope` / latched rate limiter behaviours | **Accept, documented.** Both behave as designed; the surprise is documentation, not code. Already captured in ONBOARDING §10. | -| 16 | `PlanSite` appended `in.Expired` (mtime ASC, `internal/pg/pending.go:29`) onto `Retain`'s output (mtime DESC, `internal/gc/retain.go:33-37`) without re-sorting, while the blast cap truncates from the tail (`internal/gc/plan.go`). Over-cap runs therefore reaped the **newest** abandoned sessions and starved retention entirely, while the reason string claimed "reaping oldest". Introduced by this sprint; found by the adversarial review, which reproduced it with a probe. | **Fixed** — merged set sorted newest-first before the cap; the test now asserts *which* deploys survive, not how many. | -| 17 | The blast cap ran before the dry-run branch in `ReconcileSite`, so a cap of 0 emptied the drift **report**. `drift-detect` runs dry with the same config, so a misconfigured ceiling would have reported a clean fleet. Introduced by this sprint. | **Fixed** — cap applies only to live runs; the dry run reports full drift and sets `Capped`/`CapReason` as a warning. | -| 18 | `newLiveAliasReader` validated the site *segment* but not the tail, so a format like `.freecode.camp/aliases-/production` passed boot and then fetched a key containing a literal `` — the same 404-for-every-site class P0-2 closed. | **Fixed** — boot refuses a `` token after the site segment. | -| 15 | Every drift verdict op (`drift.selfcheck`, `drift.unreadable`, `drift.aliased_missing`) is absent from `cronShapedOps` (`internal/observability/sentry.go:377`), so `alertOnDrift`'s `captureBackground(v.Op, ...)` falls through to the transient-rate tracker: threshold 3 with a 26h reset window means a nightly alert is swallowed for two nights, and the in-memory counter resets on every pod restart, so with 3 replicas it may never escalate. **A live site serving nothing could page nobody.** | **Fixed** — all four verdict ops added, pinned by a test that fails if a new one is missed. | -| 14 | `PublicURLForSite` (`internal/handler/handler.go:152`) is never assigned outside tests, so the hardcoded fallback at `deploy.go:377-382` always runs — the public URL returned to the CLI bakes `freecode.camp` and `.preview.` into the binary, while every other domain fact comes from config. There is no `ROOT_DOMAIN` setting. | **Fix with P0** (one-liner): derive the URL from the configured alias formats, or add the root domain to config. Cosmetic today, silently wrong the day the root domain or preview label changes. | +| # | Finding | Disposition | +| --- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| 1 | `LiveAliases` receives a dirname, alias format expects a slug → always 404, safety net inert in prod | **P0-2**, fix now | +| 2 | Whole gc/CLI suite runs under a format where slug == dirname, hiding the entire bug class | **P0-1**, fix now | +| 3 | `CLEANUP_BLAST_CAP` has no default → `0` → unlimited destruction | **P0-3**, fix now | +| 4 | `gc-site` moves bytes before writing the tombstone row | **P0-4**, fix now | +| 5 | `reconcile` records tombstones with hardcoded `bytes = 0` | **P0-4**, in passing | +| 6 | Abandoned `deploy.init` sessions leave unowned bytes (~18/month) | **P2**, the root cause | +| 7 | Two keyspaces with no type separation | **P1** | +| 8 | Postgres stores dirnames, registry stores slugs | **Out of scope** — migration; P1's types make it survivable | +| 9 | `runDriftReport` is called with no argv (`cmd/artemis/main.go:49`) so `driftreport ` silently ignores it; and any unrecognised subcommand falls through to `run()` (`:62`), i.e. **a mistyped subcommand starts the server** | **P0-adjacent** — operators started running these subcommands against production *this week*. A typo that boots a server, and a report that ignores the argument an operator typed, are both how a run gets misread as authoritative. Fix with P0: reject unknown subcommands, reject unexpected args. | +| 10 | A finalized deploy remains writable for the JWT's remaining TTL (up to 15 min) | **Accept, documented.** Real, but requires an authorized token holder; tightening it means invalidating the JWT at finalize, which is its own design. Record in ONBOARDING traps. | +| 11 | `outbox` has no retention — unbounded growth | **Backlog.** Small table, slow growth, no correctness impact. Needs a purge job eventually; not part of this wave. | +| 12 | Dead worker code paths | **Backlog**, cosmetic. | +| 13 | `RequireScope` / latched rate limiter behaviours | **Accept, documented.** Both behave as designed; the surprise is documentation, not code. Already captured in ONBOARDING §10. | +| 16 | `PlanSite` appended `in.Expired` (mtime ASC, `internal/pg/pending.go:29`) onto `Retain`'s output (mtime DESC, `internal/gc/retain.go:33-37`) without re-sorting, while the blast cap truncates from the tail (`internal/gc/plan.go`). Over-cap runs therefore reaped the **newest** abandoned sessions and starved retention entirely, while the reason string claimed "reaping oldest". Introduced by this sprint; found by the adversarial review, which reproduced it with a probe. | **Fixed** — merged set sorted newest-first before the cap; the test now asserts *which* deploys survive, not how many. | +| 17 | The blast cap ran before the dry-run branch in `ReconcileSite`, so a cap of 0 emptied the drift **report**. `drift-detect` runs dry with the same config, so a misconfigured ceiling would have reported a clean fleet. Introduced by this sprint. | **Fixed** — cap applies only to live runs; the dry run reports full drift and sets `Capped`/`CapReason` as a warning. | +| 18 | `newLiveAliasReader` validated the site *segment* but not the tail, so a format like `.freecode.camp/aliases-/production` passed boot and then fetched a key containing a literal `` — the same 404-for-every-site class P0-2 closed. | **Fixed** — boot refuses a `` token after the site segment. | +| 15 | Every drift verdict op (`drift.selfcheck`, `drift.unreadable`, `drift.aliased_missing`) is absent from `cronShapedOps` (`internal/observability/sentry.go:377`), so `alertOnDrift`'s `captureBackground(v.Op, ...)` falls through to the transient-rate tracker: threshold 3 with a 26h reset window means a nightly alert is swallowed for two nights, and the in-memory counter resets on every pod restart, so with 3 replicas it may never escalate. **A live site serving nothing could page nobody.** | **Fixed** — all four verdict ops added, pinned by a test that fails if a new one is missed. | +| 14 | `PublicURLForSite` (`internal/handler/handler.go:152`) is never assigned outside tests, so the hardcoded fallback at `deploy.go:377-382` always runs — the public URL returned to the CLI bakes `freecode.camp` and `.preview.` into the binary, while every other domain fact comes from config. There is no `ROOT_DOMAIN` setting. | **Fix with P0** (one-liner): derive the URL from the configured alias formats, or add the root domain to config. Cosmetic today, silently wrong the day the root domain or preview label changes. | ## Sequencing @@ -213,3 +213,29 @@ Phases map to commits by subject: `fix(gc): read live aliases...` is P0-1 + P0-2 The `reclaimable` threshold shipped at **25**, not the 50 originally sketched: with P2 collecting abandoned sessions at source the steady-state baseline is zero, so a lower bar is signal rather than noise. This document is the seed artifact for a new dossier. It does not belong in `artemis-audit-fixes`, which is at 19/20 with a pending converge. + +______________________________________________________________________ + +## Wave outcome (2026-08-17) + +The second wave on `fix/artemis-drift-at-source` closed the whole-codebase read findings: destructive write-ordering unified (row before bytes everywhere, `internal/handler/destructive_ordering_test.go` pins purge and deploy-delete), one selection function `capOldest` owns every blast-cap decision, tombstone-purge gained the cap it was missing, GitHub throttles (403 and 429, primary and secondary) classify as rate-limited and are never negative-cached, auth caches are bounded at 4096 entries, uploads carry a 10-minute request deadline, gc workflows carry explicit 30-minute execution timeouts, and ~1,500 lines of dead code left (debounce, deployflows, SetAliasCAS, GetOrFetch, emitSiteChanged, the valkey repo-request store). Adversarial review confirmed 9 findings, all closed; three were defects in this wave's own new code (429s bypassing the rate-limit classifier, a cache-bound test that asserted nothing, a wrong recovery claim in ARCHITECTURE.md). The two closing review-fix commits passed the full gate and mutation checks but received no further review round. + +## Open decisions + +Everything below waits on an operator call. The evidence is cited so the decision does not need re-derivation. + +### audit_log keyspace (finding: two keyspaces in one column) + +HTTP writers record the registry slug; the GC writers record the storage dirname (`cmd/artemis/gcwire.go:49-77` pass through the site value gc hands them, which is a dirname). The only reader that joins on the column, `DeployActors` (`internal/pg/audit.go:91`), receives the URL slug (`internal/handler/site.go:311`) — so **slug is the correct keyspace** and the GC writers are the ones to fix. No backfill of existing rows: `0006_audit_log.sql` installs BEFORE UPDATE/DELETE/TRUNCATE triggers that raise, so the table is append-only by design and rewriting history means dropping triggers on production. Recommended: convert the GC writers to slugs and record the cutover date here. + +### outbox retention + +`Enqueue` only inserts (`internal/pg/outbox.go:31`); published rows are never deleted, so the table grows without bound. Small and slow, no correctness impact. Needs a retention-window decision; the purge can ride the nightly tombstone-purge workflow once a window is chosen. + +### Slug/Dirname type split (P1, still deferred) + +Its own wave. First deliverable is the compiler-produced coercion-site list — change the type, read every resulting error — BEFORE any behaviour change, so the refactor is provably zero-runtime-effect. + +### Orphan reclaim (operator run, time-sensitive) + +The live drift report against production proposed 37 repairs (32 failed-upload prefixes, 5 lost index rows) across 9 sites. `drift.reclaimable` alerts at threshold 25, so the first nightly sweep after 1.8.0 deploys will fire until the backlog is reclaimed: `artemis reconcile --apply` per site, and the blast cap of 10 means any site holding more than 10 items needs repeat runs.