EQTY LabDocs

LLM Gateway Operations

Health checks, upgrade order, and troubleshooting guidance for the LLM Gateway.

View as Markdown

Operators should monitor the LLM Gateway as part of the gateway stack. Most user-facing failures come from a setting it shares with Viper or the Control Plane, or from the gateway database. The settings below are the chart values from LLM Gateway Configuration.

Health Checks

The gateway exposes a liveness probe at /health:

bash
curl https://gateway.example.com/health
json
{
  "status": "healthy",
  "service": "llm-gateway",
  "timestamp": "2026-10-22T12:00:00Z",
  "version": "1.2.1"
}

In a chart deployment version is the chart version; elsewhere it is dev unless the version is set.

The probe reports only that the process is serving. It does not check the database, Auth Service, plugin artifact storage, or the model providers. The gateway connects to the gateway database and applies its migrations before it starts serving, and exits if it cannot, so a pod that answers has reached the database at least once.

To exercise the full path, run an agent through Viper and confirm its exchange appears in Guardian; see Run Agents. For request rates, errors, and outcomes by decision code, use the gateway’s Prometheus metrics; see Observability.

Before Upgrades

Back up the gateway database and preserve the current gateway stack values before upgrading, and review the release notes for migrations. Roll out the LLM Gateway before the Control Plane. The gateway applies the database migrations when it starts, holding a database lock so only one replica applies them, and serves only once they finish. The startup probe allows 15 minutes for this.

A single helm upgrade rolls out both services at once, so upgrade in two steps when the release notes list migrations:

  1. Upgrade with gateway-stack.controlPlane.replicaCount: 0, and gateway-stack.controlPlane.autoscaling.enabled: false if you use autoscaling.
  2. Wait until the new gateway pods are ready.
  3. Upgrade again with your usual values.

The Control Plane is unavailable between the two upgrades. Gateway pods still on the old release keep serving agents against the migrated database until the rollout replaces them, so a migration they cannot work with fails their requests in the meantime. Watch the gateway’s error rate until every pod runs the new release. See Control Plane Operations.

On shutdown the gateway stops accepting requests and flushes its audit queue for up to five seconds. The audit and plugin queues live on the pod’s emptyDir volumes, so records still queued when a pod is deleted are lost.

Common Operational Symptoms

  • Every agent request returns 401 with sig_invalid and signature parse failed: check that signature.label is sig.
  • Requests return 401 with sig_expired: the machine running Viper has a clock behind the gateway’s by more than signature.defaultSigMaxAge plus signature.defaultClockSkew, three minutes by default, or requests take that long to arrive. A clock ahead by more than signature.defaultClockSkew returns sig_invalid with signature created in future.
  • Requests return 401 with key_not_registered, or 403 with agent_revoked: the agent’s key is not registered with this gateway, or an Organization Owner revoked the agent. See Agent Inventory.
  • Registration returns 404 registration_unavailable: registration.enabled is off, or, for an API key, authServiceBaseURL is unset.
  • A fresh registration token returns registration_invalid_token: the gateway and the Control Plane do not share the same registration token secret and issuer. In the chart both come from controlPlane.auth.
  • An API-key registration returns 502 registration_auth_unavailable: the gateway could not reach Auth Service at authServiceBaseURL, or Auth Service returned an error.
  • An agent registered with an API key is missing from its registrant’s agents in Guardian: authServiceIdentityIssuer differs from the issuer the Control Plane derives from its own Auth Service URL, so the gateway records the registrant as a different user. Remove the override, or point both services at the same Auth Service URL.
  • Requests return 502 upstream_unavailable: the gateway could not reach the provider. Check the upstream URLs and the cluster’s egress to them.
  • Requests return 413 request_too_large: the request body is larger than the 32 MiB limit, -max-request-body-bytes.
  • Requests return 426 unsupported_upgrade: the client opened a WebSocket, which the gateway does not proxy. Codex switches to HTTPS on its own; another client needs its HTTP transport configured.
  • Exchanges are missing from the Audit Trail: look for audit sink write batch failed, which means the gateway cannot write to the database and is holding exchanges in its queue, or audit queue overflow, which means the queue is full and new exchanges are being dropped.
  • A published plugin configuration does not take effect: wait one plugins.pollInterval. A plugin runtime disabled line in the startup log means plugins.enabled is off. If it still does not apply, look for plugin configuration refresh compile failed, which names the error; the gateway keeps running the previous configuration. If the gateway rejects the plugin’s artifact, check that it trusts the signer’s key.
  • Project policies and project membership never apply: look for project context resolution disabled in the startup log, and set projectContext.mode (-project-context-mode outside the chart).
  • The pod exits at startup: read its last log line. LLM Gateway failed: invalid config: names the setting at fault; failed to initialize postgres trusted registry means the database is unreachable or a migration failed, and the rest of the line names which; agent registration requires credential signer seed means registration is on without registration.credentialSigner; failed to initialize hook runtime means the active plugin configuration could not be loaded, and a new pod has no cached copy to fall back on.

Escalation Information

When escalating, include the agent ID, the provider route or registration call, the approximate time, the reason code from the error, and the X-Guardian-Correlation-Id response header from a failing request.