database: split RetryPostgres into RetryIdempotent and RetryPostgresNonIdempotent

[elovelan]

Suggested implementation of the transaction-retry pattern from the review comments on this PR:

• r3470551946 — recommending an explicit transaction (with fn taking a transaction argument like pgx.BeginTxFunc) instead of an implicit one, so there are fewer failure scenarios where a retry could duplicate committed work. Also: capture pg_current_xact_id() before commit and consult pg_xact_status() on failure to see if the commit was successful.
• r3487236671 — recommending two distinct functions: one with automatic safe retries (transactional), and one for idempotent callers where the function being retried must be idempotent.
• r3482973525 / r3483137216 — network errors and string-matched connection errors should be removed from the standalone classifier because without a transaction there's no way to know if the query landed.

What this adds

RetryPostgresNonIdempotent (safe, transactional) — wraps fn in an explicit BeginTx/Commit/Rollback and retries the whole transaction on transient errors. fn receives a *sql.Tx each attempt. This is the shape of pgx.BeginTxFunc and pgxtransactions.Transaction, and what jackc agreed should work. Use this for non-idempotent operations. Commit verification is always on (see below).

RetryIdempotent (generic, idempotent-caller) — retries fn (no transaction wrapper) on any error by default except context.Canceled/context.DeadlineExceeded. The caller vouches that fn is idempotent. Lighter-weight option for selects and idempotent DML (no transaction overhead, no commit-verification round-trip).

RetryMySQLNonIdempotent / RetrySpannerNonIdempotent (placeholders) — return errors.New("... not yet implemented") with TODOs. They share RetryNonIdempotentOptions and the fn func(*sql.Tx) error signature so the cross-backend API surface is visible.

The existing RetryPostgres is removed (this PR is unmerged). Callers should use RetryPostgresNonIdempotent (transactional) or RetryIdempotent (idempotent).

Pre-commit classifier (opt-out) + commit verification (always-on)

Pre-commit errors (from BeginTx or fn) — opt-OUT classifier isRetryablePostgresPreCommitError: pre-commit retry is always safe (the transaction rolls back), so it retries everything except context.Canceled/context.DeadlineExceeded and the permanent SQLSTATE classes (22xxx, 23xxx, 42xxx — via pgerrcode.IsDataException/IsIntegrityConstraintViolation/IsSyntaxErrororAccessRuleViolation). Unknown *pgconn.PgError codes and non-PgError errors are retried; a false positive (~200ms wasted) costs less than a false negative (spurious user-facing failure). opts.IsRetryable is additive (OR) on this classifier.

Commit errors (from (*sql.Tx).Commit) — always verified, not classified. The server is the authority on whether a commit landed (a client-side guess can't tell committed-but-response-lost from genuinely-aborted), so instead of a "retryable commit" classifier we always check pg_xact_status. Per r3470551946 and the PostgreSQL docs (which explicitly endorse pg_xact_status for "determine whether their transaction committed or aborted after the application and database server become disconnected while a COMMIT is in progress"):

1. Before each Commit, capture the transaction id: SELECT pg_current_xact_id_if_assigned()::text (NULL → read-only; the ::text cast sidesteps pgx's xid8 Go type-mapping).
2. On a commit error, query SELECT pg_xact_status($1::xid8) on a fresh connection (the original may be dead; xid8 is globally unique so this works across connections/reconnects, modulo a failover replay-lag window):
   • 'aborted' → retry (the commit didn't land).
   • 'committed' → ErrCommitted (don't retry — would duplicate; the caller reconciles/alerts).
   • NULL / 'in progress' / query errors → ErrCommitPhase (ambiguous; caller decides) - unlikely, but could happen during an asynchronous regional failover or true regional outage.
3. Read-only transactions (no xid) retry without verifying — no writes to duplicate.

A retryable commit error (verified-aborted or read-only) on exhaustion returns the original error (not ErrCommitPhase — it's known non-commit, just out of retries). ErrCommitPhase is only for inconclusive commits.

if errors.Is(err, database.ErrCommitted) {
    // Verified committed: do NOT retry (would duplicate). Reconcile/alert.
}

if errors.Is(err, database.ErrCommitPhase) {
    // Inconclusive: the commit may or may not have landed. Decide / re-check.
}

Both preserve the underlying commit error for inspection via errors.Is/errors.As.

Why no commit classifier? An earlier iteration had a "maximally conservative" commit classifier (isSafeRetryablePostgresError) that fast-pathed pgconn.SafeToRetry + class-40 commit errors to retry without verification. But commit errors are rare, and the xid-capture round-trip is already paid on every commit — so always checking pg_xact_status on a commit error (one more round-trip, only on the rare commit error) is negligible, and it removes the whole client-side guessing exercise (the 57P01/57P02/57014/57P03/53300/network/driver.ErrBadConn exclusion reasoning). Class-40 → pg_xact_status says 'aborted' → retry (same outcome, +1 round-trip); the rare SafeToRetry+failover+replay-lag case → ErrCommitPhase (recoverable). opts.IsRetryable no longer applies to commit errors — safer, since a caller shouldn't override commit verification.

Trade-off: commit verification adds one round-trip per transaction (pg_current_xact_id_if_assigned() before every commit), paid always but only used on rare commit errors. Always-on for the non-idempotent variant (where a duplicate-commit hurts most); RetryIdempotent skips it (no transaction).

Non-configurable retry budget (maxRetryAttempts = 3)

MaxAttempts is not configurable; both loops use maxRetryAttempts = 3 (in order to match database/sql's maxBadConnRetries+1). Use the context deadline to bound total time, or wrap for more. database/sql already retries driver.ErrBadConn for *sql.DB methods (immediate, up to 3 attempts); this outer loop layers on top with jitter for longer outages. *sql.Tx methods have no internal retry, so this is the only retry for tx.Exec/tx.Commit.

pgerrcode adoption

Adopted github.com/jackc/pgerrcode (canonical Postgres error-code constants, by the pgx author) to eliminate string literals and inline human-friendly-name comments: PostgresUniqueViolation/PostgresDeadlockFound use pgerrcode.UniqueViolation/pgerrcode.DeadlockDetected; isPermanentPostgresError uses pgerrcode.IsDataException/IsIntegrityConstraintViolation/IsSyntaxErrororAccessRuleViolation. Tests use pgerrcode constants throughout.

Naming

• RetryIdempotent (no transaction wrapper) — for selects and idempotent DML; the caller vouches fn is idempotent. Chosen over "unsafe" so the common case doesn't put "unsafe" all over callers' codebases.
• RetryPostgresNonIdempotent (transactional + commit verification) — for non-idempotent ops. The Tx suffix was dropped (the fn func(*sql.Tx) error parameter makes the transaction obvious at the call site).
• RetryIdempotentOptions / RetryNonIdempotentOptions (shared by the Postgres/MySQL/Spanner non-idempotent variants) — parallel to the function names.
• The internal retryTx helper keeps its name (unexported, descriptive).

Pluggable retryable cases

• RetryNonIdempotentOptions.IsRetryable is additive (OR) on the pre-commit classifier (commit errors are always verified, not classified).
• RetryIdempotentOptions.IsRetryable replaces the default (nil → retry any error except context cancellation/deadline).

Out of scope (follow-ups)

• Implementing RetryMySQLNonIdempotent / RetrySpannerNonIdempotent — placeholders only (no pg_xact_status equivalent wired; they'd fall back to ErrCommitPhase for commit errors until each backend provides a verifier).
• A sql.DB.RetryPostgresNonIdempotent method on the sql/db.go wrapper — deferred (would add a sql → database dependency).
• Integration test against a real Postgres for the verification path (simulating an ambiguous commit) — the unit tests use a mock driver that scripts pg_current_xact_id_if_assigned/pg_xact_status returns; a real-PG integration test would confirm the SQL and xid8 round-tripping.

Test plan

• TestRetryIdempotent — success, retries on any error by default (incl. 23505), custom IsRetryable short-circuits, no retry on context.Canceled, ctx cancellation, attempt exhaustion.
• TestRetryPostgresNonIdempotent — success; fn-error-then-succeed (rollback between attempts); non-retryable fn error short-circuits (23505); driver.ErrBadConn from fn retried; SafeToRetry Begin failure retried; SafeToRetry Commit failure → verified-aborted → retry; verify-aborted → retry; verify-committed → ErrCommitted; verify-unknown (NULL) → ErrCommitPhase; verify-error → ErrCommitPhase; read-only (no xid) → retry without verifying; class-40 exhaustion → verify-aborted each attempt → original error (not ErrCommitPhase); 57P01+inconclusive → ErrCommitPhase; additive opts.IsRetryable (pre-commit); ctx cancellation; attempt exhaustion. Uses a minimal database/sql/driver mock (with QueryerContext to script the verification queries).
• TestIsRetryablePostgresPreCommitError — pre-commit classifier leg coverage.
• TestErrCommitPhase / TestErrCommitted — sentinel detection; underlying error preserved; distinct from each other.
• TestRetryMySQLNonIdempotentNotImplemented / TestRetrySpannerNonIdempotentNotImplemented — placeholders return "not yet implemented".
• go build ./..., go vet ./..., go test ./database/... -short all pass.