EQTY LabDocs

Control Plane Operations

Health checks, upgrade order, and troubleshooting guidance for the Control Plane.

View as Markdown

Operators should monitor the Control Plane as part of the gateway stack. Most user-facing failures come from a dependency it calls on each request, or from a setting it shares with the LLM Gateway. The settings below are the chart values from Control Plane Configuration.

Health Checks

The Control Plane exposes a liveness probe at /health, under the API base path:

bash
curl https://control-plane.example.com/api/health
json
{
  "status": "healthy",
  "service": "control-plane",
  "timestamp": "2026-10-16T12:00:00Z",
  "version": "1.2.1"
}

In a chart deployment version is the chart version; elsewhere it is dev unless the version is set.

The probe reports only that the process is serving. It does not check the database, Auth Service, or Integrity Service. The Control Plane connects to the gateway database at startup and exits if it cannot, so a pod that stays up has reached the database at least once.

To exercise the Auth Service path, read a project’s agents with an API key that has View Project Data on a project that uses the gateway:

bash
curl -H "Authorization: Bearer $GUARDIAN_API_KEY" \
  https://control-plane.example.com/api/v1/projects/<project-uuid>/agents

A 401 means Auth Service refused the key, and a 403 that the key lacks the permission or the project doesn’t exist. A 503 with auth_service_unavailable means the Control Plane could not reach Auth Service. Successful validations and permission decisions, including denials, are cached for 30 seconds, so a new outage can take that long to appear. Outage errors themselves are not cached.

Before Upgrades

Back up the gateway database and preserve the current gateway stack values before upgrading, and review the release notes for migrations. Roll out the LLM Gateway before the Control Plane: the gateway applies the database migrations when it starts. The Control Plane does not wait for them. Its waitForRegistryViews init container only checks that the registry views exist, which they already do on an upgrade, and a single helm upgrade rolls out both services at once. When the release notes list migrations, upgrade in two steps, holding the Control Plane at zero replicas until the new gateway pods are ready; see LLM Gateway Operations.

Common Operational Symptoms

  • Every request returns 401: check auth.bearer.mode is auth_service, then Auth Service health. Outside the registration token route, an Auth Service outage also shows as 401 on routes that are not project-scoped.
  • Project-scoped views return 503 Service Unavailable: the error code names the service. Check that service’s health and that the Guardian adapter is enabled with its service URLs.
  • Submitting membership, trust, or certification evidence returns 503 with integrity_service_unavailable: check Integrity Service health and the adapter’s Integrity URL.
  • Creating a registration token returns 403: the caller lacks Register Agents. A 502 with auth_service_unavailable means Auth Service is down. A 500 saying the registration token service is not configured means no registration token secret is set.
  • Viper cannot register with a fresh token: confirm the Control Plane and the LLM Gateway use the same registration token secret and issuer.
  • A revoked membership or certification still reads as active: look for a refresh loop disabled warning in the startup log, which means the Integrity URL is missing or integrityStatusCacheTTL is not positive. Otherwise wait one integrityStatusCacheTTL interval, then check Integrity Service health and the trusted issuer DIDs.
  • Publishing a plugin fails with a missing signature: configure the publisher signer. If the gateway rejects a published plugin, check that it trusts the signer’s key. If storing a plugin secret fails, set the plugin secrets key.
  • The pod exits at startup: read its last log line. invalid config: names the setting at fault; failed to initialize discovery store means the database is unreachable.

Escalation Information

When escalating, include the agent ID or project, the affected route or Guardian page, the approximate time, the caller’s role, and the X-Guardian-Correlation-Id response header from a failing request.