Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .agents/checks/review.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ the reviewer's distillation.
- Core packages: every loop, queue, retry, and wait must be bounded. An unbounded anything in
a core package is a review-blocking defect.
- Dangerous APIs accept proof types (`statement.Classified`, `PreflightedTable`,
`AbsentTarget`, `VerifiedShadow`, `CleanWatermark`, `TableLock`) with package-private
`AbsentTarget`, `CreationRole`, `VerifiedShadow`, `CleanWatermark`, `TableLock`) with package-private
constructors — never a
raw string or bool that a caller could fabricate. Core code re-verifies its own
preconditions; it never trusts that the planner or CLI checked.
Expand Down
3 changes: 3 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,9 @@ refusal — never a silently wrong or incomplete result:
- **Unlogged tables and explicit column collations** are outside the
declarative model: converging either is a table (or column) rewrite, so
export and diff refuse rather than plan one.
- **Greenfield `CREATE TABLE` apply** is not user-reachable yet: the
executor create path exists as a library building block, but the
declarative front door does not route to it.
- **Non-table objects** — views, standalone sequences, enums, domains,
extensions, functions, triggers — are outside the declarative model,
which covers one ordinary table plus its indexes per file.
Expand Down
4 changes: 2 additions & 2 deletions SAFETY.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,8 +67,8 @@ The short version — the full rules live in [docs/tcb-model.md](docs/tcb-model.
- **Never trust callers.** Every dangerous operation re-verifies its preconditions, whoever the
requester is (CLI, planner, orchestrator). The periphery may request; the core enforces.
- **Domain types make illegal states unrepresentable.** Validating passages return proof types
with package-private constructors (today `preflight.PreflightedTable` and
`preflight.AbsentTarget`; later phases add
with package-private constructors (today `preflight.PreflightedTable`,
`preflight.AbsentTarget`, and `preflight.CreationRole`; later phases add
`VerifiedShadow`, `CleanWatermark`, and `TableLock`); dangerous APIs accept only proof types —
e.g. the planned cutover swap will accept only a `VerifiedShadow`.
- **Put a limit on everything.** Every loop bounded, every queue bounded, every retry counted,
Expand Down
2 changes: 1 addition & 1 deletion docs/capabilities.md
Original file line number Diff line number Diff line change
Expand Up @@ -171,7 +171,7 @@ Status legend: ✅ T1 (supported today) · 🟡 T2 (planned; typed refusal today
| Unlogged tables | 🟡 | Yes | Typed refusal: persistence is not modeled, converging it (`SET LOGGED`) is a full rewrite, and rendering the table as plain `CREATE TABLE` would silently change crash-safety |
| Explicit column collations | 🟡 | Yes | Typed refusal: dropping a `COLLATE` clause from a rendered baseline silently changes sort order and index semantics; a collation delta cannot converge without a rewrite |
| Columns whose default uses a sequence the column does not own | 🟡 | Yes | Typed refusal: in a desired-state model that sequence exists only inside the scratch transaction, so no derived plan can reference it. Column-owned (`serial`-style) sequences are fine |
| Greenfield `CREATE TABLE` apply (the table does not exist yet — a fresh database or a new table in a live one) | 🟡 | Yes — a `REFERENCES` clause takes a brief `SHARE ROW EXCLUSIVE` on each **referenced** live table | Planned as an owned operation. The new table itself has no readers or writers to protect; the online-safety problem is the `REFERENCES` clause, whose lock on each referenced live table queues behind long-running queries and blocks writers behind it — exactly the run-it-under-a-bounded-`lock_timeout` job this engine owns and owner tooling does not do. The absence preflight (`CheckTableAbsent`) is in place; the executor create path and front-door admission build on it. `diff --sql` already emits the statement |
| Greenfield `CREATE TABLE` apply (the table does not exist yet — a fresh database or a new table in a live one) | 🟡 | Yes — a `REFERENCES` clause takes a brief `SHARE ROW EXCLUSIVE` on each **referenced** live table | Planned as an owned operation. The new table itself has no readers or writers to protect; the online-safety problem is the `REFERENCES` clause, whose lock on each referenced live table queues behind long-running queries and blocks writers behind it — exactly the run-it-under-a-bounded-`lock_timeout` job this engine owns and owner tooling does not do. The absence preflight (`CheckTableAbsent`), the creation-privilege preflight (`CheckCreatePrivileges`), and the executor create path (`ExecuteCreate` — plain `CREATE TABLE` plus plain index builds; `PARTITION OF`, `INHERITS`, `LIKE`, `OF`, and `IF NOT EXISTS` are typed refusals at admission, while `REFERENCES` and `CONCURRENTLY` are refused upstream at desired-file parse and re-checked at admission as defense in depth) are in place; the declarative front door does not route to them yet. `diff --sql` already emits the statement |

### Types and non-table objects

Expand Down
9 changes: 9 additions & 0 deletions docs/engine-role.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,15 @@ same preflight: `wal_level = logical` (`rds.logical_replication = 1` on Aurora/R
static parameter requiring a reboot), and free `max_replication_slots` /
`max_wal_senders` headroom.

### Off-ladder: greenfield `CREATE TABLE`

Creating a new table sits outside the ladder: the table does not exist yet, so there is no
owning role to be a member of — the table is born owned by the role that creates it. The
create path's preflight (`CheckCreatePrivileges`) therefore proves exactly `CONNECT` on the
database plus `USAGE` and `CREATE` on the target schema, deliberately not the Tier 1–3
ownership membership. A missing grant is refused with the exact `GRANT` statement, whose
grantee is the engine role itself.

## Provisioning

For a target whose tables are owned by `app_owner` in schema `app`:
Expand Down
35 changes: 35 additions & 0 deletions docs/execution-model.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ and [suggest-report.md](suggest-report.md#caveats-caveats).
- [The committed prefix](#the-committed-prefix)
- [How a failure is reported](#how-a-failure-is-reported)
- [Why the prefix is safe to leave](#why-the-prefix-is-safe-to-leave)
- [Outcome codes](#outcome-codes)

## Why there is no wrapping transaction

Expand Down Expand Up @@ -241,3 +242,37 @@ statement — stopping at the first refusal or failure. Its result carries the
plan, one verdict per attempted statement, and a detail naming exactly which
planned statements committed and remain in effect: the committed prefix at
the plan level, statements instead of steps.

## Outcome codes

`executor.Codes()` enumerates the closed vocabulary below, and
`executor.OutcomeCode` maps any executor error to its entry — the same code
that reaches the JSON verdict's `code` field. Adapters render three facts
per failure — the outcome code, the failing step's position
(`SequenceStepError.Step` of `.Total`), and the failing step's SQL — and
log the raw error, whose text interpolates server prose and is not a
branching surface.

| Code | Meaning |
| --- | --- |
| `budget-lock-exceeded` | The lock was not granted within `lock_timeout`; nothing executed |
| `budget-statement-exceeded` | The statement ran past `statement_timeout` and was cancelled |
| `cancelled-externally` | The statement was cancelled from outside the executor before its budget elapsed |
| `invalid-index-own-leftover` | The failed build's own INVALID index remains; the [recovery runbook](invalid-index-recovery.md) applies |
| `invalid-index-preexisting` | An INVALID index under the requested name predates this run |
| `invalid-index-unproven` | An INVALID index may remain but the catalog state could not be proven |
| `empty-sequence` | The sequence had no steps to run |
| `unsupported-sequence-step` | A step is not a shape the sequence executor can run safely |
| `unsupported-partitioned-parent` | Partitioned-parent admission refusal |
| `not-concurrent-index-build` | The statement handed to the concurrent build executor is not a `CREATE INDEX CONCURRENTLY` |
| `unnamed-index` | The concurrent build does not name its index, so its outcome could not be verified |
| `unqualified-table` | The target table is not schema-qualified at the library boundary |
| `if-not-exists-unsupported` | `CREATE ... IF NOT EXISTS` cannot prove what its no-op would mean |
| `create-collision` | A name the create path needs is already taken on the server; re-diff the live catalog |
| `duplicate-create-name` | The desired set claims the same relation name twice; refused at admission |
| `partition-of-unsupported` | `CREATE TABLE PARTITION OF` locks the partitioned parent, which the absence proof does not cover |
| `unsupported-create-step` | A desired statement is not a shape the create path can run |
| `pool-too-small` | The pool cannot hold the build session and the verdict connection at once |
| `table-not-found` | The statement's qualified table does not exist |
| `invariant-violation` | A breach of the invariant registry; never a retry candidate |
| `execution-failed` | Fallback for a failure outside the typed set — an operational error to investigate, not a refusal to branch on |
5 changes: 3 additions & 2 deletions docs/invariants.md
Original file line number Diff line number Diff line change
Expand Up @@ -244,8 +244,9 @@ executes, any statement whose target table does not match the preflight proof it
A proof for one table can never smuggle SQL against another, and a multi-statement string can
never reach the database through the executor (pgx's simple protocol would happily run all of
it). *Enforced:* `pkg/executor` (`ExecuteNative`; `RunSequence` admission re-proves every step's
target against the preflight proof before the first step executes), `pkg/statement` (proof
construction).
target against the preflight proof before the first step executes; `ExecuteCreate` re-proves
every desired statement's target against the absence proof the same way), `pkg/statement`
(proof construction).
*Source:* adversarial review of the optimistic front door.

## Refusals and preflight (RF)
Expand Down
1 change: 1 addition & 0 deletions docs/limitations.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ with a typed refusal — never a silently wrong or incomplete result:
| Column collations | An explicit `COLLATE` on a column is not managed: converging a collation delta rewrites the column and its indexes. Export refuses a collated column — a baseline without the clause would silently change sort order and index semantics — and a collation delta (including on an added column) is a typed `diff` refusal. |
| Non-table objects | Views, materialized views, standalone sequences, enums, domains, extensions, functions, and triggers are outside the model. A serial column's owned sequence is the one exception: it round-trips through the `serial` pseudo-types — and ownership is verified through the catalog (`pg_depend`), so a hand-written `nextval` default on a standalone sequence that merely carries the serial-style name refuses rather than exporting as `serial` and silently privatizing a shared sequence. A column may *use* an unmanaged type (an enum, a domain) — the type text round-trips — but the type's definition is not managed. |
| Multiple tables per file | A desired file is single-table scoped: exactly one `CREATE TABLE` plus `CREATE INDEX` statements on it. Multi-table schemas are managed as one file per table. |
| Greenfield table creation | Not user-reachable yet: the executor create path (`executor.ExecuteCreate`) exists as a library building block, but the declarative front door does not route to it. The path runs a plain `CREATE TABLE` plus plain index builds on the table born that run, and refuses at admission — before anything executes — every clause that binds to an existing object the absence proof does not cover: `PARTITION OF`, `INHERITS`, `LIKE`, and `OF`, plus `IF NOT EXISTS`. `REFERENCES` and `CONCURRENTLY` are refused upstream at desired-file parse (`statement.ParseDesired`); the create path's admission re-checks them as defense in depth. |
| Changed index or constraint definition | A redefinition diffs to drop-and-recreate, the drop is destructive, and desired-state execution refuses any plan containing a destructive statement — the whole plan, including the harmless recreate. Run the drop deliberately first (`DROP INDEX CONCURRENTLY` directly against the database; `ALTER TABLE ... DROP CONSTRAINT` through the imperative front door), then rerun — the remaining plan converges the recreate. |

## What desired-state execution converges today
Expand Down
40 changes: 32 additions & 8 deletions docs/schemabot-integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,22 +143,46 @@ package. Landing this is one of:

### Routing the create path's refusals

The planned greenfield `CREATE TABLE` path opens with `preflight.CheckTableAbsent`, and its
proof has a rule the adapter must respect: an `AbsentTarget` is **minted inside the apply
The greenfield `CREATE TABLE` path is a fixed call order, all inside the apply session:

1. `statement.ParseDesired` — parse and validate the desired file (refuses `REFERENCES`,
`CONCURRENTLY`, qualified names).
2. `preflight.CheckCreatePrivileges` — mint the `CreationRole` proof for the target schema.
3. `preflight.CheckTableAbsent` — mint the `AbsentTarget` proof for the table name.
4. `executor.ExecuteCreate` — consume both proofs and run the set.

Both proofs share one rule the adapter must respect: they are **minted inside the apply
session and consumed there** — never serialized into `SchemaChange.Metadata`, carried across
the plan/apply boundary, or reused across retries. Absence at plan time proves nothing about
apply time; the executor re-verifies inside the session that runs the `CREATE`, the same way
ST-7 re-verifies a `PreflightedTable`.
the plan/apply boundary, or reused across retries. Absence or privilege at plan time proves
nothing about apply time; the executor re-verifies inside the session that runs the
`CREATE`, the same way ST-7 re-verifies a `PreflightedTable`.

Each refusal from the check maps to a different orchestrator action — route them, don't
retry them uniformly:
Each refusal from the preflight checks maps to a different orchestrator action — route
them, don't retry them uniformly:

| Refusal | What it means | Orchestrator action |
| --- | --- | --- |
| `ErrRelationExists` / `ErrTypeExists` (grouped by `preflight.IsNameOccupied`) | The name is already taken — this is not a create, it's a change to something that exists | Route to the diff/alter path, not to a failure state |
| `ErrSchemaNotFound` | The qualified schema does not exist on the target | Operator action (create the schema or fix the desired file); retrying cannot succeed |
| `ErrNoCreationSchema` | Unqualified name and the role's `search_path` yields no creation schema | Caller configuration: schema-qualify the name or fix the role's `search_path` |
| Duplicate-name error from the `CREATE` itself | A concurrent writer won the race after a valid proof | Re-plan from scratch — the world changed; do not blindly retry the create |
| `*preflight.PrivilegeError` (`Tier == TierCreateTable`) | The role lacks `CREATE` on the schema (or `USAGE` reaching it); the error carries the exact missing grant | Operator action: provision the named `GRANT`, then retry |

`ExecuteCreate`'s own refusals and failures carry the same routing discipline
([outcome codes](execution-model.md#outcome-codes)):

| Outcome | What it means | Orchestrator action |
| --- | --- | --- |
| `ErrDuplicateCreateName` (`duplicate-create-name`) | The desired set claims one relation name twice — including a first-choice implicit constraint-index name; refused at admission, nothing ran | Fix the desired file; retrying unchanged cannot succeed |
| `ErrPartitionOfUnsupported` (`partition-of-unsupported`) | `PARTITION OF` binds to a live parent the absence proof does not cover | Fix the desired file; out of the create path's scope |
| `ErrUnsupportedCreateStep` (`unsupported-create-step`) | A desired statement is not a shape the create path can run | Fix the desired file |
| `ErrCreateCollision` (`create-collision`) | A concurrent writer took a needed name after a valid proof | Re-diff the live catalog and re-plan — the world changed; never blindly retry the create |

A failed create is not rolled back wholesale: each step committed in its own bounded
transaction, so the steps before the failure remain
([the committed prefix](execution-model.md#the-committed-prefix)). A rerun's absence check
then refuses with `ErrRelationExists`, and the gate stays closed until the declarative
front door re-diffs the live catalog and converges the remainder — the orchestrator never
assumes the failed run left nothing behind.

## Execution-mode verdicts and direct execution

Expand Down
1 change: 1 addition & 0 deletions docs/tcb-model.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,7 @@ to obtain the type is through the function that validates it.
| `string` (user SQL) | `statement.ParseOne` / `statement.ParseOps`, then `planner.Classify` | `planner.Plan` / `planner.Decision` | CO-7 — classification consumes parsed operation descriptors |
| table name | preflight | `PreflightedTable` (carries the proven facts: PK, no FKs/views, replica identity, headroom) | ST-6, RF-* |
| table name (create target) | `preflight.CheckTableAbsent` | `AbsentTarget` (carries the resolved creation schema and the verified-free name; time-of-check — minted inside the apply session, never carried across a plan boundary, and re-verified at use the way ST-7 re-verifies `PreflightedTable`) | ST-6 for the create path |
| creating role's access (create target) | `preflight.CheckCreatePrivileges` | `CreationRole` (carries the connected role and the resolved creation schema whose CONNECT / USAGE / CREATE grants were verified; time-of-check and session-scoped, like `AbsentTarget` — a revoked grant after minting fails with the server's own error) | ST-6 for the create path |
| shadow table | full checksum pass (planned) | `VerifiedShadow` — its constructor will be private to `pkg/checksum`; the planned `cutover.Swap` will accept **only** this type | CO-1 in the type system |
| chunker low-watermark | all-checkers-clean pass (planned) | `CleanWatermark` — will be unobtainable in a pass that repaired anything | CO-2 |
| — | planned table-lock acquisition | `TableLock` token, planned as a required parameter of every mutating operation | LK-1 |
Expand Down
Loading
Loading