EQTY LabDocs

LLM Gateway Architecture

The LLM Gateway's relationships with Viper and the platform services, its endpoints, request signing, and storage.

View as Markdown

The LLM Gateway carries agent traffic; it does not manage it. This page describes the service’s boundaries and what it stores. For one request’s path through the gateway, step by step, see Gateway Basics.

Platform Relationships

ComponentProduct role
ViperRegisters the agent’s key with the gateway, uploads its state snapshots, and signs every request it sends through the gateway.
Control PlaneMints the registration tokens the gateway verifies, and publishes the plugins and configurations it runs. It reads the audit exchanges the gateway writes.
GuardianShows the gateway’s address in agent onboarding commands. Guardian reads gateway data through the Control Plane, not from the gateway.
Auth ServiceValidates an API key an agent registers with, and checks its owner’s Register Agents permission. Only when API-key enrollment is configured.
Gateway databasePostgreSQL shared with the Control Plane. The gateway creates its tables and holds the agent registry, blobs, state history, audit exchanges, and plugin state.
Plugin artifact storeObject storage the gateway downloads signed plugin artifacts from: Google Cloud Storage, an S3-compatible store, Azure Blob Storage, or a shared directory.
Model providersAnthropic, OpenAI, and the ChatGPT backend, at the upstream URLs the gateway is configured with.
Relay destinationsAny other HTTP endpoint an agent calls, except loopback, link-local, and unspecified addresses, when Viper relays the agent’s other traffic through the gateway.

Endpoints

The gateway serves its API at the root of its host. Its Prometheus metrics are served on a separate listener; see Observability.

RouteUsed forSigned
GET /healthLivenessNo
POST /v1/agents/registrationsRegistering an agent’s key with a registration token or an API keyNo
GET /v1/agents/registrations/issuerThe DID that signs registration credentials, for offline verifiers to pinNo
POST /v1/agents/credentials/renewRenewing the agent’s registration credentialYes
PUT /v1/blobs/{cid}Uploading a state snapshot, attestation, or credential before a request references itYes
/v1/anthropic_api/…, /v1/openai_api/…, /v1/chatgpt_backend/…Provider routes; the rest of the path and the query are forwarded to that providerYes
/v1/relayRelaying an agent’s other HTTP traffic to the URL in the signed X-Viper-Upstream-URLYes
GET /openapi.json, /docs/The gateway’s OpenAPI spec and API explorerNo

Registration is authenticated by the token or API key in the request instead of a signature. The full shapes are in the LLM Gateway API reference.

Request Signing

Viper signs each request with the agent’s private key using HTTP Message Signatures (RFC 9421). The signature covers the method, the target URL, a per-request nonce and timestamp, a BLAKE3 digest of the body, and the CIDs of the agent’s state snapshot and credentials. The gateway checks that:

  • the signing key is a DID registered to an agent that is not revoked
  • the signature is no older than the maximum age, one minute by default, allowing two minutes of clock skew
  • the nonce has not been seen before, and the body matches its digest
  • the state snapshot, or the state attestation credential, the request references is in the blob store and matches its CID

Blob uploads sign the method, the target URL, and the body digest. Credential renewals sign the method, the target URL, and a nonce. Replay and state checks apply only to provider and relay requests.

A request that fails any check gets a 401, or a 403 when the agent is revoked, and is not forwarded. Before forwarding an allowed request, the gateway removes the signature and X-Viper-* headers, so the provider sees the agent’s request with its own provider credentials.

Registration

An agent registers once per signing key. With a registration token, the gateway verifies the token against the secret and issuer it shares with the Control Plane; a token works once and expires after 15 minutes. With an API key, the gateway asks Auth Service whether the key is valid and carries the Register Agents scope, and whether its owner has that permission. Either way, the gateway records the DID against the agent and returns a registration credential signed with its credential signer key. Viper renews that credential with a signed request. See Register Agents.

Storage and Background Work

Everything the gateway keeps is in the gateway database. Two pieces of work happen outside the request:

  • Audit capture. Each exchange is appended to a queue on the pod’s disk and written to the database in batches, every second by default. Every 24 hours from pod start, the gateway deletes exchanges older than the retention period, 90 days by default.
  • Plugin refresh. With plugins on, the gateway reloads the active global and project configurations every 15 seconds by default, and downloads and verifies the signed artifacts they name. Plugin work that runs after a request ends goes through a second on-disk queue.

Operational Boundaries

The gateway does not hold provider API keys; agents’ own credentials pass through it. It does not mint registration tokens, manage plugin configurations, or serve the audit trail to users; the Control Plane does. It does not see traffic that bypasses Viper.

Replay protection is held in each gateway process’s memory. With more than one replica, a nonce is remembered only by the replica that served it, and the signature’s maximum age plus the clock skew bounds how long a replay could be accepted by another replica.