Performance and resource problems
Containers killed for memory, services that restart, slow or missing detections, PDF exports that fail, and the GPU and OCR paths.
- Version: 0.4
- Role: admin_user
- Type: troubleshooting
Almost every performance symptom on this appliance is one of three things: not enough memory, one service down, or a service that is up and cannot do its job because a model or a dependency is missing. The diagnostic snapshot separates them in one read.
Note
One critical finding usually explains several warnings. A host with no memory left produces one root cause and four symptoms, so work the highest-severity finding first rather than the loudest one.
Find your symptom
| What you see | Runbook |
|---|---|
| A container exited with code 137 | A container was killed for memory |
| A container keeps restarting | A container keeps restarting |
| One detection category returns nothing | One service is unhealthy |
| Detection is slow, or the whole appliance is | The machine is undersized |
| A filesystem is nearly full | A filesystem is nearly full |
| OCR returns nothing on an air-gapped appliance | OCR has no models |
| Every PDF export fails | A PDF will not render |
| Reviews never complete, or the analyst is unreachable | The analyst is unavailable |
| A file is refused, or its analysis fails | A file is refused or its analysis fails |
| Lists are empty and nothing errors | The database or Redis is unreachable |
| One dashboard action fails and nothing says why | A dashboard action fails |
Memory and capacity
A container was killed for memory
Exit code 137 is the kernel, not the service. Restarting it restores service and changes nothing about why it was chosen.
A container was killed for running out of memory
You might see: was killed for running out of memory (exit 137) · exit code 137 · an OOM kill (exit 137) means this machine needs more RAM · the detector dies under load
Before you start
Checks
- 1
Compare the host's total memory against the minimum for the licensed package
diagnostics · host.memTotalMb = 32768
- 2
Read how much memory the host has available right now
diagnostics · host.memFreeMb = 2048
- 3
Compare the killed container's declared limit with what it was using
expected · Usage sits well below the limit. Usage at the limit means the limit is the constraint, not the host.
- 4
Check whether every service running on this box is actually in use
expected · Everything running is something this deployment needs.
Causes and fixes
| Reached when | Cause | Explanation, fix and verification |
|---|---|---|
| check 1 fails | host-undersized (critical) | The full stack was measured at about 30 GiB in use, and the default profile declares roughly 38 GB of container limits with several services declaring none. Below 32 GB the kernel reclaims memory by killing the largest process, which is always a detection service loading a model. The service that dies is a symptom of the machine, not a fault in itself. Fix: Raise the machine's memory to at least 32 GB, or to 64 GB where documents and images are processed, and reboot. See The machine is below the package's RAM minimum and Sizing and requirements. Verify: diagnostics · host.memTotalMb = 32768 |
| check 2 fails | no-headroom (critical) | The machine meets the nominal minimum and has nothing left at this moment, so the next model load or document parse triggers a kill. Available memory, not total memory, is the number that predicts the next OOM. Fix: Reduce what runs concurrently, or add memory. Restarting the killed container restores service and does not change the outcome — the next peak kills something again. Verify: diagnostics · host.memFreeMb = 2048 |
| check 3 fails | limit-too-low (warning) | The host has memory and the container hit its own declared limit, so the kernel killed it while the machine was fine. This is the honest case for raising a service's limit — it is also the only case, because raising a limit on a machine with no headroom just moves the kill to a different container. Fix: Raise that service's memory limit in the deployment's compose configuration and recreate the container. Confirm the host has the headroom for the new limit before you apply it. Verify: expected · oomKilled stays false and usage settles below the new limit. |
| check 4 fails | unused-services-running (info) | Several services are opt-in — OCR, document parsing, red-teaming, the local analyst and the MCP server — and each holds memory whether or not the deployment uses it. On an undersized box, switching off what is unused is faster than a hardware change and costs nothing that is in use. Fix: Stop the services this deployment does not use and confirm nothing depends on them. Document what you switched off: a later feature request will otherwise be diagnosed as a bug. Verify: diagnostics · host.memFreeMb = 2048 |
A container keeps restarting
Separate an out-of-memory kill from a configuration the service refuses, before restarting it again.
A container keeps restarting
You might see: has restarted 12 times — it is in a crash loop · container restarting · a service comes back and dies again · the dashboard works for a minute and then errors
Before you start
Checks
- 1
Check whether the last termination was an out-of-memory kill rather than a crash
expected · oomKilled is false. When it is true, this is a sizing problem and not a crash.
- 2
Read how much memory the host has left
diagnostics · host.memFreeMb = 1024
- 3
Verify the schema is current, since a failing boot migration restarts the gateway forever
diagnostics · datastores.postgres.migrationsPending
- 4
Check whether the container refuses its own configuration at boot
expected · The service reports that it started and is listening, rather than refusing a setting.
Causes and fixes
| Reached when | Cause | Explanation, fix and verification |
|---|---|---|
| check 1 fails | repeated-oom (critical) | The container is not crashing: the kernel is killing it, and the restart policy brings it back into the same conditions. Exit code 137 is the signature. Nothing in the service's own log explains it, because the process is stopped without warning. Fix: Follow A container was killed for running out of memory. Verify: expected · oomKilled is false and the restart count stops climbing. |
| check 2 fails | host-memory-exhausted (critical) | The host has almost no memory left, so the kernel kills whichever container is largest at the moment pressure peaks. The service that dies is not the service that is at fault, which is why the restarting container appears to move around between reboots. Fix: Give the machine more RAM, or reduce what runs on it. The full stack needs 32 GB, and 64 GB for a deployment that handles documents and images. See The machine is below the package's RAM minimum. Verify: diagnostics · host.memFreeMb = 1024 |
| check 3 fails | boot-migration-loop (critical) | The gateway applies the migration chain at boot. A migration that throws takes the process with it, the container restarts, the same migration runs again, and the restart count climbs with nothing else changing. Fix: Stop the container, read one failed start end to end, and resolve the specific error. See Migrations are pending. Verify: diagnostics · datastores.postgres.migrationsPending |
| check 4 fails | configuration-refused (critical) | The service refuses to start with the configuration it was given, and says so in its first log lines. A placeholder signing secret is refused deliberately; a missing model path, an unreadable certificate and a bad database password behave the same way. The restart policy then turns a clear one-line refusal into a loop nobody reads. Fix: Fix what the log names, then start the container once and read the log again rather than leaving it to restart. If the refusal is about the signing secret, see The JWT signing secret is a placeholder. Verify: expected · The service reports that it is listening and the restart count stops climbing. If that does not apply: Attach the support bundle and the first twenty lines of one failed start. |
The machine is undersized
The default compose profile declares roughly 42 GB of container memory limits, eleven services declare none, and the full sixteen-service stack was measured at about 30 GiB in use. Below the package minimum the kernel evicts whichever detection service is largest exactly when load peaks.
The machine has less RAM than the licensed package needs
You might see: This box has 16.0 GB of RAM; the package needs 32 GB · ML services will be OOM-killed under load · memory is 8192 MB; 16 GB is the designed size · the appliance runs but services keep being evicted
Before you start
Checks
- 1
Compare total memory against the 32 GB minimum for the full stack
diagnostics · host.memTotalMb = 32768
- 2
For a deployment that processes documents and images, compare against the 64 GB recommendation
diagnostics · host.memTotalMb = 65536
- 3
Check whether the undersizing has already cost a service
expected · No container has been killed for memory.
Causes and fixes
| Reached when | Cause | Explanation, fix and verification |
|---|---|---|
| check 1 fails | below-minimum (critical) | The full stack was measured at about 30 GiB in use. Below 32 GB the machine cannot hold the detection services and their models at the same time, so the kernel evicts whichever is largest when pressure peaks. The product works in a demo and fails under the first real load, which is why this is worth fixing before go-live rather than after. Fix: Raise the machine's memory to at least 32 GB and reboot. If the hardware cannot be changed, run only the core gateway and the three mandatory detection services and switch the rest off deliberately — a 16 GB box is a development configuration, not a supported production one. See Sizing and requirements. Verify: diagnostics · host.memTotalMb = 32768 |
| check 2 fails | below-recommendation (warning) | 32 GB carries the stack; documents, images and OCR add the largest transient allocations in the product, and 64 GB is what keeps them from competing with the detectors. A GPU is recommended for that work and never required. Fix: Raise the machine to 64 GB before enabling document and image analysis for a whole organisation, or keep those features to a pilot group until you can. Verify: diagnostics · host.memTotalMb = 65536 |
| check 3 fails | already-losing-services (critical) | The undersizing is no longer theoretical: a container has been killed or is restarting. Any detection that should have run while a detector was down did not run, and nothing retries it. Fix: Raise memory first, then review the detections for the affected window — a detector that was down did not fail closed. See A container was killed for running out of memory. Verify: expected · No container reports an out-of-memory kill and restart counts are stable. |
A filesystem is nearly full
An update needs room for a second copy of every image, and the appliance's state directory fails silently when the root filesystem fills — so a full disk shows up first as settings that do not persist.
A filesystem is nearly full
You might see: is 94% full (6 GB free of 100 GB) · an update needs room for a second copy of every image · no space left on device · the update fails part way through
Before you start
Checks
- 1
Read the free space on the root filesystem
diagnostics · host.disks[/].freeGb = 20
- 2
Read the free space where Docker keeps its images and volumes
diagnostics · host.disks[/var/lib/docker].freeGb = 40
- 3
Read how large the CID database has grown
diagnostics · datastores.postgres.dbSizeMb = 50000
Causes and fixes
| Reached when | Cause | Explanation, fix and verification |
|---|---|---|
| check 1 fails | root-filling (critical) | The appliance writes its state, its logs and the update spool to the root filesystem. When it fills, writes that the product wraps in a log-and-continue handler fail silently — the setup state, the licence and update intents all behave as if they were saved and are not. Fix: Free space or grow the disk, then restart the gateway container so it re-reads its state. Check the journal and any log files the site collects: an unrotated log is the usual reason an appliance that ran for months fills up. Verify: diagnostics · host.disks[/].freeGb = 20 |
| check 2 fails | image-store-filling (critical) | An update downloads a second copy of every image before it switches over, so an appliance needs roughly the size of the release set free at the moment it updates. Without that room the update fails part way through and rolls back, which reads as a broken update rather than a full disk. Fix: Remove images the appliance no longer runs, then update. Do not delete volumes to make room: they hold the database, the detections and the appliance identity. Verify: diagnostics · host.disks[/var/lib/docker].freeGb = 40 |
| check 3 fails | retention-too-long (warning) | The detection and event history grows with use, and nothing prunes it by default. On a busy appliance the database becomes the largest single consumer of the disk, and the first symptom is an update that will not fit rather than a database that complains. Fix: Reduce the retention window for detections and event logs to what your compliance obligation actually requires, and confirm the size falls afterwards. Export first if the records must be kept outside the appliance. Verify: diagnostics · datastores.postgres.dbSizeMb = 50000 |
Services
One service is unhealthy
Reachable-and-unhealthy is the interesting case, and on an air-gapped appliance it usually means a model that did not load. Read the dependency first for the analyst and the MCP server: they report themselves unhealthy when something they need is down, and fixing them wastes the window.
One service is unreachable or reports itself unhealthy
You might see: hap-guard-v2 is unreachable: no response · ocr-service answered but is not healthy · ml-detector is unreachable · one detector is down and the others are fine
Before you start
Checks
- 1
Read whether the named service answered its health endpoint at all
expected · reachable is true. When it is false the failure is network or container level, not model level.
- 2
Read the container state for the named service
expected · state is running, restarts are low and oomKilled is false.
- 3
For a service that answers and reports itself unhealthy, read the error it returns
expected · The error names what did not initialise — a model file, a device, a dependency.
- 4
For the analyst and the MCP server, check what they depend on before blaming them
expected · The dependency is healthy, so the unhealthy verdict is about this service.
Causes and fixes
| Reached when | Cause | Explanation, fix and verification |
|---|---|---|
| check 1 fails | not-reachable (critical) | The health endpoint did not answer at all, so the problem is below the application: the container is not running, it is still starting, or the gateway is pointed at the wrong address. A service that is not deployed at all is absent from the snapshot rather than unreachable, so an unreachable row means something that should be there is not answering. Fix: Start the container and wait for its healthcheck. If it is running, check that the URL the gateway uses matches the port the container listens on — several services listen on a port that differs from the host publication, and a hand-edited environment file is the usual way that drifts. Verify: expected · reachable and healthy are both true. |
| check 2 fails | container-not-running (critical) | The container is exited, restarting or dead. Detection services load their models into memory at start, so a machine under memory pressure kills these first and the failure looks like a broken service rather than an undersized appliance. Fix: Follow A container was killed for running out of memory when oomKilled is true, and A container keeps restarting otherwise. Verify: expected · state is running and the restart count is stable. |
| check 3 fails | model-did-not-load (warning) | The service answers and reports itself unhealthy, which almost always means a model that did not load. On an air-gapped appliance the usual reason is an image built without its model files: the service starts, discovers the files are absent, and reports the failure rather than crashing. Fix: Read the error the health endpoint returns, then follow the runbook for that service. For the OCR service, see The OCR service has no models on an air-gapped appliance. Restarting a service whose model files are absent changes nothing. Verify: expected · healthy is true and no error is present. If that does not apply: Attach the support bundle, which carries the log tail for every service. |
| check 4 fails | dependency-unhealthy (warning) | Some services report themselves unhealthy because something they need is down, not because they are broken. Fixing the wrong one wastes the maintenance window, and the dependency's own row already says so. Fix: Resolve the dependency first — the database for the MCP server, the MCP server and the local model for the analyst — then re-read the snapshot before touching this service. Verify: expected · The dependency is healthy and this service follows. |
OCR has no models
An image built before the model check shipped can carry the engine and not its models: healthy on a networked machine, which fetches them on demand, and fatal air-gapped. Restarting a service whose model files are absent changes nothing.
Note
When rebuilding the OCR image, CACHEBUST must stay declared in the compose file's build args
— compose silently drops an undeclared build argument, so the pre-download layer is served
from cache forever and a failed download looks permanent.
The OCR service has no models on an air-gapped appliance
You might see: HTTP 503: det/rec models not found · ocr-service answered but is not healthy · image analysis returns no text on the appliance and works on the dev box · OCR works when the machine has internet and fails when it does not
Before you start
Checks
- 1
Read whether the OCR service reports itself healthy
diagnostics · services[ocr-service].healthy = true
- 2
Rule out an out-of-memory kill, which produces an unhealthy OCR service for a different reason
expected · oomKilled is false.
Causes and fixes
| Reached when | Cause | Explanation, fix and verification |
|---|---|---|
| check 1 fails | models-absent-from-image (warning) | OCR models are baked into the image at build time. Constructing the engine downloads only the orientation classifier on current versions — detection and recognition arrive on first use — so a build that stopped at construction produced an enormous image holding a few megabytes of models. That image is healthy on a networked machine, which fetches the rest on demand, and fatal on an air-gapped one, which cannot. The build now forces a real prediction and refuses the image when detection and recognition are absent, so this affects images built before that check. Fix: Replace the OCR image with one from a build that passes the model check; there is nothing to repair inside a running container, because the files were never in it. When rebuilding, pass a cache-busting build argument — the pre-download layer is otherwise served from cache and a failed download looks permanent. Verify: diagnostics · services[ocr-service].healthy = true If that does not apply: Attach the support bundle and the image digest the snapshot reports for ocr-service. |
| check 2 fails | ocr-oom (critical) | The OCR service holds the largest transient allocations in the product, so on an undersized machine it is the first thing the kernel kills. The result — an unhealthy OCR service and image analysis that returns nothing — looks identical to a missing model until you read the kill flag. Fix: Follow A container was killed for running out of memory. 64 GB is the recommendation for a deployment that processes documents and images. Verify: expected · oomKilled is false and the service reports healthy. |
GPU
A GPU is recommended, never required, and only for document, OCR and image-heavy work. There
are two GPU paths and nothing else uses one: the CUDA compose overlay, and the gpu profile's
Chandra-based OCR service. An appliance with no GPU is not misconfigured, and no finding fires
for its absence — if OCR is slow on a CPU-only box, that is the documented behaviour, not a
fault.
Reports and analysis
A PDF will not render
Rendering is a separate service the gateway calls over REPORT_RENDERER_URL. With it absent,
every PDF export fails while everything else on the page keeps working — which is why it looks
specific to one button.
A PDF or report will not render
You might see: Failed to render compliance scorecard PDF · Failed to render detections PDF · Failed to render executive summary PDF · Failed to render regulations PDF
Before you start
Checks
- 1
Read whether the report renderer answered
diagnostics · services[report-renderer].reachable = true
- 2
Read whether the renderer reports itself healthy
diagnostics · services[report-renderer].healthy = true
- 3
Check whether only the large exports fail
expected · The small export succeeds.
- 4
For a compliance dossier, read whether the render was reconciled against the record
expected · The renderer reported its rendered ids.
Causes and fixes
| Reached when | Cause | Explanation, fix and verification |
|---|---|---|
| check 1 fails | renderer-not-reachable (warning) | PDF rendering is a separate service the gateway calls over REPORT_RENDERER_URL. With that service absent or unreachable, every PDF export fails and everything else on the page keeps working — which is why the failure looks specific to one button.Fix: Start the report renderer and confirm the gateway's URL matches the port it listens on. Follow One service is unreachable or reports itself unhealthy if it will not come up. Verify: diagnostics · services[report-renderer].reachable = true |
| check 2 fails | renderer-unhealthy (warning) | A renderer that answers and reports itself unhealthy usually failed to initialise something it needs — a font set or a headless browser. It will keep answering and keep failing. Fix: Read the renderer's error and its container logs, then restart or rebuild the service. Verify: diagnostics · services[report-renderer].healthy = true |
| check 3 fails | renderer-resource-limited (warning) | Rendering holds the whole document in memory. A large date range or a wide estate produces a document that exceeds the container's memory limit, and the kernel kills the renderer rather than the request failing cleanly. Fix: Export a narrower range, or raise the renderer's memory limit. Check whether the container was OOM-killed — see A container was killed for running out of memory. Verify: expected · The PDF downloads. |
| check 4 fails | dossier-unreconciled (warning) | A compliance dossier is evidence. The product will not hand over a rendered document it could not check against the frozen record, because an unchecked report is worse than no report. The dossier JSON is unaffected and stays downloadable. Fix: Fix the renderer first, then re-render the dossier. Use the JSON in the meantime — it is the same record. Verify: expected · The PDF is produced and the download is allowed. If that does not apply: Attach the support bundle, the report and date range you exported, and the renderer's container log tail. |
The analyst is unavailable
The analyst and its MCP server are opt-in and off by default
(RISK_ANALYST_SERVICE_ENABLED). When enabled they are the largest thing on the box, the first
thing memory pressure kills, and the slowest to come back.
A file is refused or its analysis fails
Size caps, unsupported formats and a document parser behind a circuit breaker all refuse in the same place. The circuit breaker is why the failure can look instant.
A file is refused, or its analysis fails
You might see: No file uploaded (expected multipart field "file") · Image size exceeds maximum allowed size of 10MB · Unsupported file type. Upload a PDF, DOCX or TXT. · legacy and macro-enabled spreadsheet formats cannot be safely redacted
Before you start
Checks
- 1
Read whether the format is one the product accepts at all
expected · The format is accepted on that surface.
- 2
Read the size limit for that surface
expected · The file is under the limit named in the message.
- 3
Check whether the client was doing a chunked or resumable upload
diagnostics · posture.resumableUploadPolicy = "block"
- 4
Read whether document analysis is switched on for this deployment
expected · The service is enabled.
- 5
Read the document parser's own health
diagnostics · services[document-parser].healthy = true
Causes and fixes
| Reached when | Cause | Explanation, fix and verification |
|---|---|---|
| check 1 fails | format-not-accepted (info) | Some formats are refused on purpose rather than partially handled. A macro-enabled spreadsheet cannot be redacted safely, and a redaction that silently missed content would be worse than a refusal. Fix: Save the file in an accepted format — a spreadsheet as .xlsx without macros, a document as PDF, DOCX or TXT — and upload it again.Verify: expected · It is accepted. |
| check 2 fails | over-size-limit (info) | Each surface caps what it will buffer, because analysis holds the whole file in memory and an unbounded upload is an out-of-memory kill for whichever detection service is largest. Fix: Split or compress the file, or use the document path rather than the image path for large scans. Verify: expected · It is accepted. |
| no check points here | resumable-refused-by-policy (warning) | No path buffers across requests, so a chunked or resumable upload cannot be inspected. Rather than passing it through uninspected, all three paths refuse it by default and the extension records a visible "not inspected" event. Reassembly — buffer by upload id, inspect at finalize, re-fragment — is not implemented. Fix: Have the user upload the file in a single request, or accept the risk explicitly by changing the policy. Do not read a refusal here as a product fault: it is the fail-closed choice working. Verify: diagnostics · posture.resumableUploadPolicy = "block" |
| check 4 fails | parser-disabled (warning) | Document analysis is a separate service and can be switched off, for instance on a minimal development stack. The endpoint answers 503 with a distinct message so it is not confused with a service that is enabled and down. Fix: Enable the document parser and restart the gateway so it re-reads the setting. See File redaction. Verify: expected · The analysis runs. |
| check 5 fails | parser-unhealthy (critical) | The gateway holds a circuit breaker in front of the parser: once it has failed repeatedly the gateway stops calling it and answers "temporarily unavailable" immediately, which is why the failure can look instant. Fix: Follow One service is unreachable or reports itself unhealthy. Verify: diagnostics · services[document-parser].healthy = true If that does not apply: Attach the support bundle and the file's type and size. Do not attach the document itself unless policy allows it to leave the estate. |
Related
- Health matrix — every service and what breaks while it is down.
- Diagnostic snapshot — where these findings come from.
- Resource sizing — the supported profiles.
Last updated on