EQTY LabDocs

PDFGen Operations

Health checks, upgrade notes, and troubleshooting guidance for PDFGen.

View as Markdown

Operators should monitor PDFGen alongside Governance Service, which is its only caller. Most report failures come from Auth Service refusing or failing to sign, from the timestamp authority, or from a report too large for the pod. The settings below are the chart values from PDFGen Configuration.

Health Checks

PDFGen has no ingress. Reach it from inside the cluster, or forward its port. These examples use the release name guardian-platform:

bash
kubectl port-forward svc/guardian-platform-eqty-pdfgen 8080:8080 --namespace guardian
curl http://localhost:8080/health/ready
json
{
  "status": "ok",
  "checks": {
    "typst": { "ok": true, "version": "0.14.2", "embedded": true },
    "fonts": { "ok": true, "count": 320 },
    "tmp_dir": { "ok": true, "path": "tmp" },
    "timestamp": { "ok": true, "enabled": true, "message": "RFC 3161 timestamping configured" },
    "typst_package_cache": { "ok": true, "path": "/opt/app-root/src/.cache/typst/packages" }
  }
}

/health/live reports only that the process is serving. /health/ready, which the chart’s startup and readiness probes use, also checks that renders can work: at least one font on TYPST_FONT_PATHS, a writable tmpDir, and that the template package cache directory exists. A failing check returns 503 with the same checks under detail, as {"detail": {"status": "unready", "checks": …}}. The failing check has "ok": false, and tmp_dir and typst_package_cache add a message. The timestamp check always passes; enabled reports whether reports get a timestamp.

Neither probe calls Auth Service or the timestamp authority. To exercise the full path, export a report from Guardian as an Organization Owner or Project Owner; see Reports.

Before Upgrades

PDFGen keeps no data, so an upgrade needs no backup or migration. Review the release notes for changed environment variables, especially any you set through extraEnvVars. On shutdown PDFGen stops accepting requests and finishes the renders in flight. The chart sets no termination grace period, so Kubernetes’ default of 30 seconds applies; an export still running after it fails. During a rollout, an export can also fail with a connection error while a stopping pod is still listed. Either way, the user exports it again.

Common Operational Symptoms

An export that fails in PDFGen returns 502 with “Failed to generate policy report” from Governance Service. The Governance Service log line Failed to generate policy report from pdfgen service carries PDFGen’s status and detail in its error field, such as report request failed [500]: {"detail":"signing failed: …"}. PDFGen logs a request line with the same detail.

  • report request failed: with a connection error, and no status: Governance Service cannot reach PDFGen. Check that its pods are ready and that governance-service.config.pdfgenServiceUrl is unset or names the PDFGen service.
  • report request failed: with timeout awaiting response headers: PDFGen did not answer within five minutes of the manifest being sent, because every render slot stayed busy or the pod is stuck. Check its readiness and logs, and add CPU or replicas; see PDFGen Configuration.
  • [500] with signing failed: and returned HTTP 501: the platform key’s algorithm has no PDF certificate profile; see Key Configuration. Other statuses from preflight cert fetch, certificate endpoint, or signing API, or any of them unreachable, point at Auth Service and its key provider; see Auth Service Operations.
  • [500] with signing failed: timestamp request to … failed, timestamp authority returned HTTP, timestamp authority refused the request, or invalid timestamp response: the timestamp authority is unreachable, refused the request, or returned a token that failed verification. Allow egress to it, or set config.timestampUrl to "".
  • [500] with signature does not verify against the cached certificate, returned a certificate for a different key than the cached one, or signing key rotated since the certificate was fetched: the platform key rotated during the render. Export again.
  • [500] with bound signing: and returned HTTP 404: with config.signingBound on, PDFGen could not fetch the signing certificate. If the quoted body starts with Cannot GET, Auth Service predates bound signing; set config.signingBound: false. If it says Signing key not found or disabled, the platform key is missing; see Key Configuration.
  • [500] with typst compile exceeded PDFGEN_TYPST_TIMEOUT: the report took longer to lay out than the timeout. Raise PDFGEN_TYPST_TIMEOUT through extraEnvVars, or give the pod more CPU.
  • [422]: PDFGen rejected the manifest, and detail says why. If it names PDFGEN_MAX_REQUEST_BYTES, raise that limit through extraEnvVars, with ephemeral-storage to match; otherwise escalate with the detail.
  • [400] with unsupported paper size or unknown format: a script sent a paper other than us-letter, us-legal, or a4, or a format other than zip or pdf. Governance Service passes both through unchecked.
  • Exports wait a long time, then succeed: every render slot is busy. Governance Service waits up to five minutes for PDFGen to answer. Add CPU or replicas; see PDFGen Configuration.
  • [500] with the compile worker failed (signal: 9: the pod ran out of memory, and the kernel ended that render. Set or raise the memory limit for the size of report being exported. If memory runs out outside a render, the pod restarts as OOMKilled instead.
  • The pod never becomes ready: curl /health/ready and read the failing check. A fonts count of 0 means config.typstFontPaths was changed. A failing tmp_dir names the error in its message: the volume is full, or PDFGEN_TMP_DIR was overridden through extraEnvVars to a path outside it.
  • The pod exits at startup: its last line, error: …, names the environment variable at fault; see PDFGen Configuration.
  • Adobe Acrobat reports a report’s signature as invalid: the platform key is secp256k1; see PDFGen Overview.

Escalation Information

When escalating, include the project, the applied policy, the approximate time, the exporting user’s role, whether they exported from Guardian or with an API key and its scopes, the paper size and format, and the detail from the Governance Service log line. Do not send the report or its attachments unless support asks for them through an approved channel.