Backup and Restore

Overview

Governance evidence must survive infrastructure failure, accidental deletion, and operator error. Backup frequency, retention, encryption, and restore testing are customer operational responsibilities; keep restore procedures aligned with your business continuity requirements. This guide defines what to protect and the recommended mechanism for each data store, so a backup strategy can be designed alongside the deployment rather than after it.

Back up before upgrades, configuration changes, and maintenance that could affect stored governance evidence.

What to Back Up

The PostgreSQL databases and the object storage contents reference each other (attachments are stored in object storage, their metadata in the database). Back them up on the same schedule so a restore produces a consistent pair.

Auth Service’s database also records the public key of each OpenBao key. With OpenBao, snapshot its storage on the same schedule as PostgreSQL and restore the two as a pair; a mismatched pair breaks signing.

PostgreSQL

How you back up PostgreSQL depends on whether the platform uses an external database or the bundled one.

External Database (Recommended for Production)

When using a managed database as described in the External Database guide, use the provider’s native backup features:

• AWS RDS: enable automated backups with at least 7 days retention (--backup-retention-period 7), which also enables point-in-time recovery. Take a manual snapshot before every platform upgrade:

  bash

  aws rds create-db-snapshot \
    --db-instance-identifier guardian-platform \
    --db-snapshot-identifier governance-pre-upgrade-$(date +%Y%m%d)

  An instance created from the AWS RDS guide before 1.3 is named governance-platform, and upgrading the chart doesn’t rename it; use that identifier.

• Azure Database for PostgreSQL - Flexible Server: configure the backup retention period (7-35 days) and choose geo-redundant backup storage for cross-region recovery

• GCP Cloud SQL: enable automated backups and point-in-time recovery on the instance

The platform databases (governance, IntegrityServiceDB, and guardian_gateway with the Gateway Stack) live on the same instance, so instance-level snapshots cover all of them.

Bundled PostgreSQL

The bundled in-cluster PostgreSQL has no built-in backup mechanism, which is one of the main reasons production deployments should use an external database. If you must run the bundled instance, schedule logical dumps of all its databases to object storage:

kubectl exec statefulset/guardian-platform-postgresql \
  --namespace guardian -- \
  bash -c 'PGPASSWORD="$(cat "$POSTGRES_PASSWORD_FILE")" pg_dumpall -U postgres --clean' \
  > guardian-platform-$(date +%Y%m%d).sql

A release installed before 1.3 keeps statefulset/governance-platform-postgresql in the governance namespace, including after the 1.3 upgrade:

kubectl exec statefulset/governance-platform-postgresql \
  --namespace governance -- \
  bash -c 'PGPASSWORD="$(cat "$POSTGRES_PASSWORD_FILE")" pg_dumpall -U postgres --clean' \
  > governance-platform-$(date +%Y%m%d).sql

Run this from a scheduled job (for example, a Kubernetes CronJob) and copy the output to a versioned bucket. Volume snapshots of the PostgreSQL persistent volume via your CSI driver are a useful complement, but only logical dumps are portable across storage classes and clusters.

Object Storage

Both the governance bucket/container and the integrity rootstore should have:

• Versioning enabled, so overwrites and deletions are recoverable
• Replication to a second region or account for disaster recovery
• Lifecycle rules that expire old noncurrent versions to control cost

Provider specifics:

• AWS S3: aws s3api put-bucket-versioning --bucket <bucket> --versioning-configuration Status=Enabled, plus a replication rule to a bucket in another region
• Azure Blob Storage: enable blob versioning and soft delete on the storage account; use object replication or RA-GRS redundancy
• GCS: enable object versioning; use dual-region buckets or a transfer job to a second bucket

Key Management

DID signing keys in AWS KMS, Azure Key Vault, and Google Cloud KMS are not exportable, so they cannot be backed up in the conventional sense. The strategy is preventing loss instead:

• AWS KMS: keys are created with a deletion window (7 days by default in the platform configuration), so a scheduled deletion can be canceled. Alert on ScheduleKeyDeletion events via CloudTrail
• Azure Key Vault: enable soft-delete and purge protection on the vault, so deleted keys are recoverable for the retention period and cannot be force-purged
• Google Cloud KMS: key versions are created with a scheduled-destroy period (24 days by default, set by auth-service.config.keyManagement.gcp_kms.scheduledDestroyDays), so a destroyed version can be restored within it. Alert on DestroyCryptoKeyVersion events in Cloud Audit Logs
• OpenBao Transit: Auth Service requires keys with deletion and plaintext backup disabled, and cannot delete them itself. Snapshot OpenBao’s storage with PostgreSQL, as described above, and restrict who can administer the Transit mount

Losing signing keys does not lose governance data, but it breaks signing for existing DIDs. Auth Service schedules key deletion itself when it deletes a DID key, so treat deletion alerts from any other principal as critical.

Configuration and Secrets

• Keep values.yaml in version control; it contains no credentials
• Keep secrets.yaml in an approved secret manager (never in version control)
• Keep the rendered release (helm get manifest output) with each backup
• Record the deployed chart version and image tags from the release manifest with each backup, so a restore can redeploy the matching versions
• In restricted environments, keep the mirrored images and chart packages with the release records, as the Air-Gapped Install describes, so a restore can install without reaching an external registry

The governance worker encryption key (encryption-key in platform-governance-worker) deserves special attention: Auth Service encrypts the service account credentials it stores with it, and a restored database cannot decrypt them without the same key. Ensure it is captured in the secret manager copy of secrets.yaml, and verify the stored copy matches the in-cluster secret after any rotation.

Recommended Schedule

Tighten frequency and retention to match your organization’s recovery point objectives and compliance requirements.

Restore Order

Restore components in this order, so services agree on identity, storage, keys, and data state:

1. Provision infrastructure (cluster, network, database instance, buckets, vault access), and in restricted environments the registry holding the mirrored images and chart packages
2. Restore secrets and configuration (secrets.yaml values, Kubernetes secrets, CA bundles)
3. Restore the PostgreSQL databases from the chosen snapshot or dump, and OpenBao storage from the matching snapshot
4. Restore or re-point object storage (replicated bucket or restored versions) from the same point in time as the database
5. Deploy the chart using the values and chart version recorded in the release manifest
6. Validate sign-in, project access, attachments, reports, indicators, lineage, and certificates before returning the platform to normal use

Restore Testing

A backup that has never been restored is not a backup. Quarterly, restore the latest database snapshot and a sample of object storage content into an isolated environment, deploy the chart against it, and run the validation checks above. Record the time taken: that is your actual recovery time objective.

Official Documentation

• Helm Chart Examples

Related

• External Database — managed PostgreSQL setup with provider-native backups
• Upgrade — the procedure that starts with a backup
• Rollback — the procedure these backups make possible
• Release Manifest — the chart and image versions to record with each backup
• Air-Gapped Install — the mirrored images and chart packages a restricted environment restores from
• Integrity Service Operations — why integrity records and blobs restore together