Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 45 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,16 +38,17 @@ The features provided by this plugin are:
> in the object store, restore is currently tested only with full backup recovery
> to the latest backup. Reports on more advanced recovery attempts are welcome.

This plugin is currently only compatible with S3 object storage.
This plugin is compatible with S3 and Azure Blob object storage.

The following storage solutions have been tested and confirmed to work with
this implementation:

- [MinIO](https://min.io/) – An S3-compatible object storage solution.
- [Azure Blob Storage](https://azure.microsoft.com/products/storage/blobs).

Known missing features:

- support for other object storage solutions (GCS, Azure),
- support for other object storage solutions (GCS),
- backups from replicas,
- proper support for private certificate authorities.

Expand Down Expand Up @@ -110,7 +111,7 @@ Use `kubectl` to apply the manifest for the latest commit in the `main` branch:
<!-- x-release-please-start-version -->
```sh
kubectl apply -f \
https://github.com/operasoftware/cnpg-plugin-pgbackrest/releases/download/v0.6.1/manifest.yaml
https://github.com/operasoftware/cnpg-plugin-pgbackrest/releases/download/v0.6.0/manifest.yaml
```
<!-- x-release-please-end -->

Expand Down Expand Up @@ -194,6 +195,47 @@ spec:
cpu: "2"
```

For Azure Blob storage, use the `azureCredentials` field instead of
`s3Credentials`. The `bucket` field is used as the Azure container name:

```yaml
apiVersion: pgbackrest.cnpg.opera.com/v1
kind: Archive
metadata:
name: azure-store
spec:
configuration:
repositories:
- destinationPath: /
bucket: backups
azureCredentials:
# keyType defaults to "shared". Use "sas" to pass a SAS token as the key.
account:
name: azure
key: AZURE_STORAGE_ACCOUNT
key:
name: azure
key: AZURE_STORAGE_KEY
compression: zst
```

When targeting an Azure-compatible endpoint (for example the
[Azurite](https://github.com/Azure/Azurite) emulator), set an explicit
`endpointURL` and use path-style addressing via `uriStyle: path`, since these
endpoints expose the storage account name in the URL path rather than the host:

```yaml
azureCredentials:
uriStyle: path
account:
name: azure
key: AZURE_STORAGE_ACCOUNT
key:
name: azure
key: AZURE_STORAGE_KEY
endpointURL: azurite:10000
```

> [!IMPORTANT]
> Unlike Barman, pgBackRest requires object storage to be accessible over HTTPS. While
> it's possible to disable key verification and use self-signed keys, using HTTP
Expand Down
68 changes: 55 additions & 13 deletions config/crd/bases/pgbackrest.cnpg.opera.com_archives.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
annotations:
controller-gen.kubebuilder.io/version: v0.21.0
controller-gen.kubebuilder.io/version: v0.19.0
name: archives.pgbackrest.cnpg.opera.com
spec:
group: pgbackrest.cnpg.opera.com
Expand Down Expand Up @@ -53,18 +53,6 @@ spec:
- lz4
- zst
type: string
createStanza:
default: OnFirstArchive
description: |-
CreateStanza controls when the pgBackRest stanza is created. `OnFirstArchive`
(default) creates it on the first WAL archive if it does not exist yet, so
archiving works without a prior backup. `OnBackup` creates it only when a backup
runs. `Disabled` never creates it automatically.
enum:
- OnFirstArchive
- OnBackup
- Disabled
type: string
data:
description: |-
The configuration to be used to backup the data files
Expand Down Expand Up @@ -121,6 +109,56 @@ spec:
repository, including all data needed to properly connect and authenticate with
a selected object store.
properties:
azureCredentials:
description: The credentials to use to upload data to Azure
Blob Storage
properties:
account:
description: The reference to the secret containing
the storage account name
properties:
key:
description: The key to select
type: string
name:
description: Name of the referent.
type: string
required:
- key
- name
type: object
key:
description: The reference to the secret containing
the account shared key or SAS token
properties:
key:
description: The key to select
type: string
name:
description: Name of the referent.
type: string
required:
- key
- name
type: object
keyType:
default: shared
description: KeyType specifies the type of key used,
either "shared" (default) or "sas"
enum:
- shared
- sas
type: string
uriStyle:
description: |-
Azure Repository URI style, either "host" (default) or "path".
The "path" style is required when targeting Azure-compatible
endpoints such as the Azurite emulator.
enum:
- host
- path
type: string
type: object
bucket:
minLength: 1
type: string
Expand Down Expand Up @@ -293,6 +331,9 @@ spec:
uriStyle:
description: S3 Repository URI style, either "host"
(default) or "path".
enum:
- host
- path
type: string
type: object
required:
Expand Down Expand Up @@ -679,6 +720,7 @@ spec:
procMount denotes the type of proc mount to use for the containers.
The default value is Default which uses the container runtime defaults for
readonly paths and masked paths.
This requires the ProcMountType feature flag to be enabled.
Note that this field cannot be set when spec.os.name is windows.
type: string
readOnlyRootFilesystem:
Expand Down
14 changes: 14 additions & 0 deletions internal/cnpgi/operator/specs/secrets.go
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,13 @@ import (

// CollectSecretNamesFromCredentials collects the names of the secrets
func CollectSecretNamesFromCredentials(pgbackrestCredentials *pgbackrestApi.PgbackrestCredentials) []string {
// A repository with both credential types set is an invalid configuration
// that is rejected upstream when building the pgBackRest command/env vars;
// don't grant RBAC access to secrets from an ambiguous configuration here.
if pgbackrestCredentials.HasConflictingCloudProviders() {
return nil
}

var references []*machineryapi.SecretKeySelector
if pgbackrestCredentials.AWS != nil {
references = append(
Expand All @@ -33,6 +40,13 @@ func CollectSecretNamesFromCredentials(pgbackrestCredentials *pgbackrestApi.Pgba
pgbackrestCredentials.AWS.SecretAccessKeyReference,
)
}
if pgbackrestCredentials.Azure != nil {
references = append(
references,
pgbackrestCredentials.Azure.Account,
pgbackrestCredentials.Azure.Key,
)
}

result := make([]string, 0, len(references))
for _, reference := range references {
Expand Down
96 changes: 48 additions & 48 deletions internal/pgbackrest/api/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -97,8 +97,43 @@ type S3Credentials struct {
Region string `json:"region,omitempty"`

// S3 Repository URI style, either "host" (default) or "path".
// TODO: Enforce values via Enum like iin compression.
// +optional
// +kubebuilder:validation:Enum=host;path
URIStyle string `json:"uriStyle,omitempty"`
}

// AzureKeyType is the type of key used for Azure credentials
type AzureKeyType string

const (
// AzureKeyTypeShared uses a storage account shared key
AzureKeyTypeShared = AzureKeyType("shared")
// AzureKeyTypeSAS uses a shared access signature token
AzureKeyTypeSAS = AzureKeyType("sas")
)

// AzureCredentials is the type for the credentials to be used to upload
// files to Azure Blob Storage.
type AzureCredentials struct {
// KeyType specifies the type of key used, either "shared" (default) or "sas"
// +optional
// +kubebuilder:default:=shared
// +kubebuilder:validation:Enum=shared;sas
KeyType AzureKeyType `json:"keyType,omitempty"`

// The reference to the secret containing the storage account name
// +optional
Account *machineryapi.SecretKeySelector `json:"account,omitempty"`

// The reference to the secret containing the account shared key or SAS token
// +optional
Key *machineryapi.SecretKeySelector `json:"key,omitempty"`

// Azure Repository URI style, either "host" (default) or "path".
// The "path" style is required when targeting Azure-compatible
// endpoints such as the Azurite emulator.
// +optional
// +kubebuilder:validation:Enum=host;path
URIStyle string `json:"uriStyle,omitempty"`
}

Expand All @@ -107,6 +142,17 @@ type PgbackrestCredentials struct {
// The credentials to use to upload data to S3
// +optional
AWS *S3Credentials `json:"s3Credentials,omitempty"`

// The credentials to use to upload data to Azure Blob Storage
// +optional
Azure *AzureCredentials `json:"azureCredentials,omitempty"`
}

// HasConflictingCloudProviders reports whether more than one cloud provider
// credential block is configured. A single pgBackRest repository can only
// target one storage type, so having both set is an invalid configuration.
func (c PgbackrestCredentials) HasConflictingCloudProviders() bool {
return c.AWS != nil && c.Azure != nil
}

// PgbackrestRetention an object containing the backup retention time for all backup
Expand Down Expand Up @@ -320,23 +366,6 @@ type PgbackrestRepository struct {
Retention *PgbackrestRetention `json:"retention,omitempty"`
}

// StanzaCreatePolicy controls when the pgBackRest stanza is created.
// +kubebuilder:validation:Enum=OnFirstArchive;OnBackup;Disabled
type StanzaCreatePolicy string

const (
// StanzaCreateOnFirstArchive creates the stanza the first time a WAL is archived,
// if it does not exist yet, so archiving works without waiting for a backup.
StanzaCreateOnFirstArchive StanzaCreatePolicy = "OnFirstArchive"

// StanzaCreateOnBackup creates the stanza only when a backup runs (the legacy behavior).
StanzaCreateOnBackup StanzaCreatePolicy = "OnBackup"

// StanzaCreateDisabled never creates the stanza automatically; it must be created
// out of band.
StanzaCreateDisabled StanzaCreatePolicy = "Disabled"
)

// PgbackrestConfiguration is the configuration of all pgBackRest operations
type PgbackrestConfiguration struct {
Repositories []PgbackrestRepository `json:"repositories"`
Expand Down Expand Up @@ -368,41 +397,12 @@ type PgbackrestConfiguration struct {
// this parameter is omitted
// +optional
Stanza string `json:"stanza,omitempty"`

// CreateStanza controls when the pgBackRest stanza is created. `OnFirstArchive`
// (default) creates it on the first WAL archive if it does not exist yet, so
// archiving works without a prior backup. `OnBackup` creates it only when a backup
// runs. `Disabled` never creates it automatically.
// +kubebuilder:validation:Enum=OnFirstArchive;OnBackup;Disabled
// +kubebuilder:default=OnFirstArchive
// +optional
CreateStanza StanzaCreatePolicy `json:"createStanza,omitempty"`
}

// GetCreateStanzaPolicy returns the configured stanza creation policy, defaulting to
// OnFirstArchive when unset.
func (c *PgbackrestConfiguration) GetCreateStanzaPolicy() StanzaCreatePolicy {
if c.CreateStanza == "" {
return StanzaCreateOnFirstArchive
}
return c.CreateStanza
}

// ShouldCreateStanzaOnArchive reports whether the WAL archive path should create the
// stanza when it is missing.
func (c *PgbackrestConfiguration) ShouldCreateStanzaOnArchive() bool {
return c.GetCreateStanzaPolicy() == StanzaCreateOnFirstArchive
}

// ShouldCreateStanzaOnBackup reports whether the backup path should create the stanza.
func (c *PgbackrestConfiguration) ShouldCreateStanzaOnBackup() bool {
return c.GetCreateStanzaPolicy() != StanzaCreateDisabled
}

// ArePopulated checks if the passed set of credentials contains
// something
func (credentials PgbackrestCredentials) ArePopulated() bool {
return credentials.AWS != nil
return credentials.AWS != nil || credentials.Azure != nil
}

// AppendAdditionalRestoreCommandArgs adds custom arguments as pgbackrest restore command-line options
Expand Down
6 changes: 6 additions & 0 deletions internal/pgbackrest/api/config_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -157,6 +157,12 @@ var _ = Describe("Pgbackrest credentials", func() {
AWS: &S3Credentials{},
}.ArePopulated()).To(BeTrue())
})

It("can check when Azure credentials are set", func() {
Expect(PgbackrestCredentials{
Azure: &AzureCredentials{},
}.ArePopulated()).To(BeTrue())
})
})

var _ = Describe("Pgbackrest retention", func() {
Expand Down
30 changes: 30 additions & 0 deletions internal/pgbackrest/api/zz_generated.deepcopy.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading