---
title: "LLM Gateway Operations"
description: "Health checks, upgrade order, and troubleshooting guidance for the LLM Gateway."
source: "Guardian Platform"
---
Operators should monitor the LLM Gateway as part of the gateway stack. Most user-facing failures come from a setting it shares with Viper or the Control Plane, or from the gateway database. The settings below are the chart values from [LLM Gateway Configuration](./configuration.mdx).

## Health Checks

The gateway exposes a liveness probe at `/health`:

```bash
curl https://gateway.example.com/health
```

```json
{
  "status": "healthy",
  "service": "llm-gateway",
  "timestamp": "2026-10-22T12:00:00Z",
  "version": "1.2.1"
}
```

In a chart deployment `version` is the chart version; elsewhere it is `dev` unless the version is set.

The probe reports only that the process is serving. It does not check the database, Auth Service, plugin artifact storage, or the model providers. The gateway connects to the gateway database and applies its migrations before it starts serving, and exits if it cannot, so a pod that answers has reached the database at least once.

To exercise the full path, run an agent through Viper and confirm its exchange appears in Guardian; see [Run Agents](../user-guide/run-agents.mdx#check-that-it-worked). For request rates, errors, and outcomes by decision code, use the gateway's Prometheus metrics; see [Observability](../deployment/observability.mdx#agent-gateway-telemetry).

## Before Upgrades

Back up the gateway database and preserve the current gateway stack values before upgrading, and review the release notes for migrations. Roll out the LLM Gateway before the Control Plane. The gateway applies the database migrations when it starts, holding a database lock so only one replica applies them, and serves only once they finish. The startup probe allows 15 minutes for this.

A single `helm upgrade` rolls out both services at once, so upgrade in two steps when the release notes list migrations:

1. Upgrade with `gateway-stack.controlPlane.replicaCount: 0`, and `gateway-stack.controlPlane.autoscaling.enabled: false` if you use autoscaling.
2. Wait until the new gateway pods are ready.
3. Upgrade again with your usual values.

The Control Plane is unavailable between the two upgrades. Gateway pods still on the old release keep serving agents against the migrated database until the rollout replaces them, so a migration they cannot work with fails their requests in the meantime. Watch the gateway's error rate until every pod runs the new release. See [Control Plane Operations](../control-plane/operations.mdx#before-upgrades).

On shutdown the gateway stops accepting requests and flushes its audit queue for up to five seconds. The audit and plugin queues live on the pod's `emptyDir` volumes, so records still queued when a pod is deleted are lost.

## Common Operational Symptoms

- Every agent request returns `401` with `sig_invalid` and `signature parse failed`: check that `signature.label` is `sig`.
- Requests return `401` with `sig_expired`: the machine running Viper has a clock behind the gateway's by more than `signature.defaultSigMaxAge` plus `signature.defaultClockSkew`, three minutes by default, or requests take that long to arrive. A clock ahead by more than `signature.defaultClockSkew` returns `sig_invalid` with `signature created in future`.
- Requests return `401` with `key_not_registered`, or `403` with `agent_revoked`: the agent's key is not registered with this gateway, or an Organization Owner revoked the agent. See [Agent Inventory](../user-guide/agent-inventory.mdx#revoke-and-restore-an-agent).
- Registration returns `404 registration_unavailable`: `registration.enabled` is off, or, for an API key, `authServiceBaseURL` is unset.
- A fresh registration token returns `registration_invalid_token`: the gateway and the Control Plane do not share the same registration token secret and issuer. In the chart both come from `controlPlane.auth`.
- An API-key registration returns `502 registration_auth_unavailable`: the gateway could not reach Auth Service at `authServiceBaseURL`, or Auth Service returned an error.
- An agent registered with an API key is missing from its registrant's agents in Guardian: `authServiceIdentityIssuer` differs from the issuer the Control Plane derives from its own Auth Service URL, so the gateway records the registrant as a different user. Remove the override, or point both services at the same Auth Service URL.
- Requests return `502 upstream_unavailable`: the gateway could not reach the provider. Check the upstream URLs and the cluster's egress to them.
- Requests return `413 request_too_large`: the request body is larger than the 32 MiB limit, `-max-request-body-bytes`.
- Requests return `426 unsupported_upgrade`: the client opened a WebSocket, which the gateway does not proxy. Codex switches to HTTPS on its own; another client needs its HTTP transport configured.
- Exchanges are missing from the Audit Trail: look for `audit sink write batch failed`, which means the gateway cannot write to the database and is holding exchanges in its queue, or `audit queue overflow`, which means the queue is full and new exchanges are being dropped.
- A published plugin configuration does not take effect: wait one `plugins.pollInterval`. A `plugin runtime disabled` line in the startup log means `plugins.enabled` is off. If it still does not apply, look for `plugin configuration refresh compile failed`, which names the error; the gateway keeps running the previous configuration. If the gateway rejects the plugin's artifact, check that it trusts the signer's key.
- Project policies and project membership never apply: look for `project context resolution disabled` in the startup log, and set `projectContext.mode` (`-project-context-mode` outside the chart).
- The pod exits at startup: read its last log line. `LLM Gateway failed: invalid config:` names the setting at fault; `failed to initialize postgres trusted registry` means the database is unreachable or a migration failed, and the rest of the line names which; `agent registration requires credential signer seed` means registration is on without `registration.credentialSigner`; `failed to initialize hook runtime` means the active plugin configuration could not be loaded, and a new pod has no cached copy to fall back on.

## Escalation Information

When escalating, include the agent ID, the provider route or registration call, the approximate time, the reason code from the error, and the `X-Guardian-Correlation-Id` response header from a failing request.

## Related

- [LLM Gateway Configuration](./configuration.mdx) — the settings these checks depend on
- [Observability](../deployment/observability.mdx) — gateway metrics and OpenTelemetry settings
- [Audit Trail](../user-guide/gateway-audit-trail.mdx#denied-exchanges) — the reason codes agents see for denied requests
- [Upgrade](../deployment/upgrade.mdx) — the full platform upgrade process
- [Control Plane Operations](../control-plane/operations.mdx) — checks for the management side of the gateway