EQTY LabDocs

Rollback

Rollback guidance for Guardian Platform upgrades.

View as Markdown

Rollback restores the platform to a previous approved release when an upgrade cannot be accepted. Plan rollback before upgrading.

Before Rolling Back

Confirm whether the failed upgrade changed platform data. Services apply their database migrations when the new release starts, and a Helm rollback does not reverse them. If the previous release cannot run against the migrated schema, or the release notes identify non-reversible changes, restore the databases from the pre-upgrade backup and follow the release-specific recovery plan.

Rolling Back from 1.3 to 1.2.x

Returning a 1.3 installation to 1.2.x requires restoring the pre-upgrade backup. A helm rollback alone leaves the platform partly unusable, because three 1.2.x services can’t work against the schema 1.3 migrated:

Service1.2.x against the 1.3 schemaNeeds the restore on its own
Auth ServiceRunsNo
Governance ServiceRunsNo
LLM GatewayStarts, but refuses every request that names a project with project_resolver_error, because 1.3 drops the table of project references it readsYes
Integrity ServiceDoesn’t start. It refuses the migrations it doesn’t know: migration ... was previously applied but is missing in the resolved migrationsYes
Control PlaneStarts, but its agent and project routes fail, because 1.3 drops the table of project references they read. With browser sign-in on, sign-in also fails, because 1.3 drops the tables that hold its identities and sessionsYes

Because these services need it, restore the whole platform from the one pre-upgrade backup, following the Rollback Flow: the PostgreSQL databases, OpenBao storage if Auth Service signs with OpenBao, and object storage from the same point in time. Anything recorded after the upgrade is lost.

  • Roll back the chart and its values together. A 1.2.x Control Plane turns on browser sign-in by default, which 1.3 no longer configures, so it doesn’t start with 1.3 values. helm rollback restores both.
  • Protect the pre-upgrade backup. 1.3 replaces every header value outside an allowlist of standard protocol and provider headers with [REDACTED] in stored LLM Gateway audit records, including signatures, API keys, content digests, and X-Guardian-Correlation-Id. No down migration brings them back. The backup still holds those values, so restrict access to it as you would to the credentials themselves, and delete it under your retention policy once the upgrade is accepted.
  • OpenBao installations return to a 1.2.1 limitation. After the restore, a 1.2.1 Control Plane and LLM Gateway reject project membership credentials signed with an OpenBao (P-256) key. See Known Issues.
  • The same limits apply during the upgrade. Once the LLM Gateway has migrated, a 1.2.x Control Plane fails its agent and project routes and a 1.2.x LLM Gateway pod refuses requests that name a project, until the last 1.2.x pod is replaced, which is why the gateway stack upgrades in two steps. See LLM Gateway Operations.

Rollback Flow

  1. Stop or pause affected user workflows if required by the incident plan.

  2. If the release-specific plan requires a restore, restore before rolling back the chart, so no service runs against data that is about to be replaced:

    1. Scale the platform’s service Deployments to zero, leaving the bundled PostgreSQL running if you use it.
    2. Restore secrets, then the databases, OpenBao storage, and object storage from the same backup, as Restore Order steps 2–4 describe.

  3. Restore the previous chart version and values, either with helm rollback to the last good revision or by re-running the install command with the previous --version and values files:

    bash
    helm history guardian-platform --namespace guardian
helm rollback guardian-platform <revision> --namespace guardian

    A release installed before 1.3 is governance-platform in the governance namespace; use those names instead.

    Rolling back the chart also restores the previous replica counts, which brings the services back on the restored data.

  4. Verify sign-in, project access, attachments, reports, indicators, lineage, and certificates.

  5. Record the incident, release versions, evidence, and customer decision.

Support Notes

Keep the release manifest, previous values, current values, and validation evidence with the support record. Do not continue retrying an upgrade until the rollback cause is understood.