feat: backups taken from a standby via pgBackRest multi-host TLS (draft PoC for #103)

[melancholictheory]

This is a draft proof of concept for #103 (support backups taken from a standby). It is meant to make the design concrete and get direction on the operational pieces before investing further. It is not proposed as merge-ready.

Problem

CloudNativePG's default backup target is prefer-standby, so backups are dispatched to a replica to keep I/O off the primary. Today the plugin configures pgBackRest for a single local instance, so a backup that lands on a standby fails at stanza-create with pgBackRest exit 56 (unable to find primary cluster). Multi-instance clusters are forced back to target: primary.

As discussed on #103, simply skipping stanza-create on a standby is not enough: pgbackrest backup itself needs a primary among its configured hosts to run pg_backup_start/pg_backup_stop, so the real fix is to give pgBackRest a connection to the current primary (multi-host).

What this PR does

• Adds an opt-in PgbackrestConfiguration.BackupStandby field (y/prefer/n, default disabled) that gates the feature.
• When a backup runs on a standby and the feature is enabled, configures the current primary as a second pgBackRest host over TLS (--pg2-host ... --pg2-host-type tls ... --pg2-host-cert-file/key-file/ca-file ... --pg2-path/user/socket-path) and passes --backup-standby=<mode>.
• Gives stanza-create the same primary peer (so it can initialise from a standby) without --backup-standby.
• Discovers the primary from cluster.Status.CurrentPrimary (already used in the WAL path) and resolves it to the primary pod's IP. The pod is read uncached (added to the manager cache DisableFor, so it needs only get pods, not a cluster-wide Pod informer), and a pods get RBAC rule is declared. Falls back to a normal local backup when this instance is the primary or no primary is known.
• Adds PgbackrestServerOptions (the pgbackrest server TLS-server invocation) as the client-visible half of the design.
• Unit tests for the generated backup / stanza-create options (including the disabled no-op path and both y and prefer modes), the server options, the primary-peer decision, and resolveStandbyTopology (via a fake client).

Deliberately out of scope (needs maintainer direction)

The operational half from #103 is not wired here, because it depends on decisions I did not want to pre-empt:

• Running the long-lived pgbackrest server process in each sidecar (lifecycle, port exposure).
• Provisioning the pgBackRest TLS certificates (a dedicated chain vs reusing existing material) and the tls-server-auth identity.

Because of that, this cannot yet run a real standby backup end to end without those certificates and the server in place, and there is no e2e test flipping test/e2e/.../backup/fixtures.go off target: primary yet. The cert and server paths are constants and a command builder, documented as the contract the plugin expects.

Testing

go build ./..., go vet, gofmt, and the unit tests (go test ./internal/..., with envtest for the controller package) all pass. New unit tests assert the exact generated pgBackRest invocations, including that the disabled path is byte-for-byte unchanged.

Open questions

Same as #103. I would appreciate direction on the TLS server lifecycle, certificate provisioning, and primary discovery (pod IP vs a dedicated headless service) before extending this.

[Agalin]

Do we actually need this field? CNPG defines in the Backup resource where it should run. If anything, we might want a simple boolean value that would explicitly enable/disable the feature and define additional dependencies.

I believe we already discussed that ideally there would be separate flags for enabling/disabling cusotm service and certificate SAN injection. That would give a yaml structure like this I believe:

spec:
  backupStandby:
    enabled: true
    injectSAN: true
    injectService: true

[melancholictheory]

Agreed on the shape. Booleans read better than mirroring pgBackRest's y/prefer/n, and it matches the opt-outs we landed on in #103.

On "do we need the field": I'd split it into two decisions that live in different places.

• Enabling the feature is cluster-level and ahead of time. The service and SAN injection happens at cluster reconcile (the mutate-cluster hook), before any Backup exists, so it needs an Archive-level switch. That's backupStandby.enabled, with injectSAN and injectService as per-piece opt-outs. This PoC doesn't do the injection yet (only the multi-host --backup-standby wiring), so that part lands with the restructure.
• Routing a backup to a standby is per-backup, and CNPG already owns it via the Backup/ScheduledBackup target. So you're right that the plugin shouldn't read y/prefer/n from the Archive. It can detect at backup time that it's running on a replica (because target: prefer-standby routed it there), configure the second host and pass --backup-standby, and skip it on the primary.

So I'd drop BackupStandby y/prefer/n for:

spec:
  configuration:
    backupStandby:
      enabled: true        # provision TLS multi-host support (service + SAN)
      injectSAN: true      # opt out to manage serverAltDNSNames yourself
      injectService: true  # opt out to manage the headless service yourself

and derive --backup-standby from whether the backup actually landed on a standby.

One thing to confirm: pgBackRest's prefer mode gets redundant if we derive from target. target: prefer-standby already means "standby if available, else primary" at the CNPG level, so we'd pass --backup-standby=y only when we land on a replica. Does that match what you want?

Happy to rework the PoC onto this, and fold in the service and SAN injection, unless you'd shape it differently.

[Agalin]

Yes, that seems correct. At pgbackrest level we either deal with running on primary (backup from primary) or running on standby (deploy from standby). We can assume CNPG operator routes the request to a correct node.