EQTY LabDocs

Observability

Operational signals and monitoring expectations for the Guardian Platform.

View as Markdown

The Guardian Platform should be monitored as a multi-service application. Operators should watch both service health and user-facing workflow success.

Signals to Monitor

  • Pod health and restart behavior.
  • Service readiness and health checks.
  • Sign-in and authorization failures.
  • Database connectivity and migration errors.
  • Object storage access for attachments.
  • Key-management access for signing and credential workflows.
  • Event ingestion and indicator evaluation failures.
  • Report generation and certificate issuance failures.

Agent Gateway Telemetry

The Agent Gateway serves Prometheus metrics at /metrics on a listener separate from agent traffic, :10001 by default; change it with the -metrics-listen flag or LLM_GATEWAY_METRICS_LISTEN. The counters cover requests per agent and route (gateway_http_requests_total), 4xx and 5xx responses per status code and agent (gateway_http_request_errors_total), and outcomes by traffic kind, route type (proxied, local_deny, upstream_error), decision code, and upstream host (gateway_request_outcomes_total). The gateway-stack chart’s Service exposes only the API port, so make the metrics port reachable to Prometheus with pod annotations or a separate Service; see the chart’s Prometheus metrics guide.

Pod annotations for annotation-based Prometheus discovery of the default listener, in the gateway-stack values (nest under gateway-stack: when installing through guardian-platform):

yaml
llmGateway:
  podAnnotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "10001"
    prometheus.io/path: "/metrics"

It exports traces, logs, and metrics over OTLP, configured with the standard OpenTelemetry variables such as OTEL_EXPORTER_OTLP_ENDPOINT. Set OTEL_TRACES_EXPORTER, OTEL_LOGS_EXPORTER, or OTEL_METRICS_EXPORTER to none to turn one off. Telemetry carries no request or response bodies; the Audit Trail is the record of what agents sent.

Optional Monitoring Chart

The deployment repository’s governance-ops chart adds Grafana dashboards and Prometheus alert rules for a platform release, and HTTP endpoint probes when prometheus-blackbox-exporter is installed. It expects kube-prometheus-stack with the Grafana dashboard sidecar enabled, and NGINX Ingress Controller metrics for the request and error panels. Install it separately from the platform, naming the platform release:

bash
helm install governance-ops ./charts/governance-ops \
  --namespace monitoring \
  --set targetRelease=guardian-platform

Use it when it matches the customer monitoring stack, or translate the same signals into the customer’s standard tooling.

Escalation Context

When escalating an operational issue, include the affected component, release version, approximate time, project or user impact, and whether the issue affects one workflow or the full platform.