Provider and egress problems
Credentials that will not save, chat requests that fail before the model, rate limits, and an appliance that cannot reach the provider at all.
- Version: 0.4
- Role: admin_user
- Type: troubleshooting
Everything between "the key is correct" and a completion streaming back. Find the symptom in the table, then work the runbook under it.
Four different things fail in ways that look identical from the chat window: the catalogue has no model, no credential resolves, the credential is refused by the provider, and the provider is never reached at all. The runbooks below separate them in that order, because each one is cheaper to check than the next.
Find your symptom
| What you see | Runbook |
|---|---|
| The provider or model list is empty | No providers are in the catalogue |
| A credential will not save, or its test fails | A credential is rejected |
| Chat fails instantly, naming a provider, model or credential | A chat request fails before the model |
| Everything times out, and the network team says the network is open | The appliance cannot get out |
| Requests are refused as too frequent, or a quota is exhausted | Rate limits and quotas |
An API client gets 401 with a cid_key_ token | A gateway API key is refused |
The catalogue
No providers are in the catalogue
Providers and models are seeded data, not code. With no rows, no model id resolves and every completion fails at the first lookup — before any credential or network path is involved.
No LLM providers are in the catalogue, so every completion fails
You might see: No LLM providers are in the catalogue · chat cannot resolve a model · the model dropdown is empty · every completion fails with a model error
Before you start
Checks
- 1
Verify the gateway can reach the database, since the catalogue is read from it rather than hard-coded
diagnostics · datastores.postgres.reachable = true
- 2
Verify no migrations are pending, so the provider and model tables have their current shape
diagnostics · datastores.postgres.migrationsPending
- 3
Ask the gateway for the model catalogue
GET /models
Causes and fixes
| Reached when | Cause | Explanation, fix and verification |
|---|---|---|
| check 3 fails | catalogue-never-seeded (critical) | Provider and model rows come from the database seeder, not from code. A box seeded with seeding switched off entirely has an empty catalogue, and chat cannot resolve a model for any request. Nothing is corrupt: the rows were never inserted. Fix: Run the seeder once against this database with sample accounts disabled, so the catalogue lands without creating demo users, then reload. The seed inserts providers and models only; credentials stay yours to add. Verify: GET /models |
| check 1 fails | database-unreachable (critical) | The gateway cannot reach the database, so every list is empty rather than erroring visibly. The model catalogue is simply the list whose emptiness stops the product working. Fix: Follow PostgreSQL or Redis is unreachable, then re-check the catalogue. Verify: diagnostics · datastores.postgres.reachable = true |
| check 2 fails | schema-behind (critical) | Pending migrations mean the provider or model tables are missing columns the current code selects, so the query fails or returns nothing and the catalogue reads as empty. Fix: Apply the pending migrations and restart the gateway. See Migrations are pending. Verify: diagnostics · datastores.postgres.migrationsPending |
Credentials
A credential is rejected
Three refusals share this screen and have nothing else in common: the endpoint validator refuses a URL it will not store, the assignment rule refuses a credential attached to both a tenant and a group, and the provider refuses the key itself. Only the third is about the key.
Note
Provider keys are stored as v1:-prefixed AES-256-GCM ciphertext and are never readable back
through the dashboard. Retyping the key is the only way to confirm what is stored. Values
written before the transformer existed pass through unchanged, so a legacy row on an old
deployment can still be plaintext.
A provider credential will not save, or its test fails
You might see: Invalid Anthropic API key · Test failed to run · Failed to create credential · Failed to update credential
Before you start
Checks
- 1
For a credential that carries an endpoint (Azure OpenAI, or any self-hosted endpoint), read the exact rejection text
expected · The endpoint is an absolute http(s) URL that does not resolve into link-local space.
- 2
Check what the credential is assigned to
expected · Exactly one of them is set.
- 3
Check whether the appliance can reach the provider at all before blaming the key
diagnostics · appliance.egress
- 4
Read what the credential test actually reported
expected · The test succeeds.
Causes and fixes
| Reached when | Cause | Explanation, fix and verification |
|---|---|---|
| check 1 fails | endpoint-refused-by-validator (warning) | The gateway validates every outbound endpoint before it will store it (config/outbound-url.ts): it must parse as a URL, it must be http or https, and it must not resolve into link-local or cloud instance-metadata space. The last rule is an SSRF guard, and it is the one that surprises people pointing CID at a model server on the same host.Fix: Give the endpoint as an absolute https://host[:port]/path URL that resolves to a routable address. For a model server beside the appliance, use its container or LAN name rather than a link-local address.Verify: expected · The form accepts it. |
| check 2 fails | assignment-not-exclusive (info) | A credential belongs to one tenant or to one tenant group, never to both — the same XOR constraint the database enforces on ai_credentials. A form that carries both values is refused before anything is written.Fix: Clear whichever of the two assignments you did not mean, then save. Verify: expected · The form accepts it. |
| check 3 fails | cannot-reach-provider (critical) | The provider call leaves the appliance like any other outbound request. Where egress is blocked or must traverse a proxy that is not configured, the provider never answers and the failure is reported as a credential problem. Fix: Follow The appliance cannot get out. Verify: diagnostics · appliance.update.channelReachable = true |
| check 4 fails | key-rejected-upstream (critical) | The provider itself refused the key. The usual reasons are a key from a different organisation or project, a key that has been rotated at the provider, or an Azure deployment whose endpoint and key belong to different resources. Fix: Re-issue the key at the provider and paste it again. Keys are stored as AES-256-GCM ciphertext and are never readable back through the dashboard, so retyping is the only way to confirm what is stored. For Azure OpenAI, set the endpoint URL on the same credential — the provider cannot be called without it. Verify: expected · The test succeeds. If that does not apply: Attach the support bundle and the exact test message; do not attach the key. |
A chat request fails before the model
The gateway resolves the model row, then a credential, then calls the provider. A failure at each step reads the same to the user and needs a different fix — and an inactive model breaks every client still naming it, including clients that worked yesterday.
A chat request fails before the model is reached
You might see: Provider 'openai' not found · Model 'gpt-4o' not found · Model 'gpt-4o' is not active · Model not found for provider
Before you start
Checks
- 1
Check that the provider and model catalogue is populated at all
expected · Providers and models are listed.
- 2
Read whether the model the caller named exists and is active
expected · The model is listed and active.
- 3
Check that a credential resolves for this caller and provider
expected · One active credential resolves.
- 4
Test the credential that resolves
expected · The test succeeds.
Causes and fixes
| Reached when | Cause | Explanation, fix and verification |
|---|---|---|
| check 1 fails | catalogue-empty (critical) | Providers and models are seeded data, not code. With no rows, no model id can resolve and every completion fails at the first lookup. Fix: Follow No providers are in the catalogue. Verify: expected · Providers and models are listed. |
| check 2 fails | model-inactive-or-absent (warning) | The gateway resolves the model row before it resolves a credential, and it refuses an inactive model rather than silently substituting one. A model deactivated in the catalogue therefore breaks every client still naming it, including clients that worked yesterday. Fix: Re-activate the model under Models, or change the client to a model that is active. A client that hard-codes a model id needs that id to stay active. Verify: expected · The completion starts streaming. |
| check 3 fails | no-credential-resolves (critical) | Credentials resolve tenant-first and fall back to the tenant's group. A tenant in no group, or a group whose credential was deactivated, resolves to nothing and the request fails before any provider call is made. Fix: Add an active credential for that provider on the tenant, or add the tenant to a group that already holds one. See Credentials. Verify: expected · The completion starts streaming. |
| check 4 fails | credential-rejected (critical) | A credential resolves and the provider refuses it, so the failure surfaces on the chat path rather than on the credential page where it was configured. Fix: Follow A provider credential will not save, or its test fails. Verify: expected · Both succeed. If that does not apply: Attach the support bundle and the request's model id, tenant and timestamp. |
Egress
The appliance cannot get out
Name resolution, a TCP connection on 443, and an authenticated request through the site proxy fail independently. A key that is never delivered fails exactly like a wrong key, which is why the egress check belongs before the credential check whenever nothing at all is working.
The appliance cannot get out — one or more egress tests failed
You might see: egress tests failed · the appliance cannot reach the registry · ETIMEDOUT after 5000ms · proxy returned 407 Proxy Authentication Required
Before you start
Checks
- 1
Verify the appliance actually ran its outbound probes
diagnostics · appliance.egress
- 2
Read the name-resolution results for the hosts this appliance must reach
diagnostics · appliance.dns
- 3
Read whether an outbound proxy is configured on the appliance
diagnostics · appliance.proxy.configured = true
- 4
Read whether the release channel answered on the last attempt
diagnostics · appliance.update.channelReachable = true
- 5
Check whether a TLS-inspection device is presenting its own certificate to the appliance
expected · The issuer is a public certificate authority, not your organisation's inspection CA.
Causes and fixes
| Reached when | Cause | Explanation, fix and verification |
|---|---|---|
| check 1 fails | probes-disabled (info) | The outbound probes are switched off for this deployment, so the snapshot carries no evidence either way and no egress finding can fire. This is a deliberate setting for a box that must make no outbound request at all. Fix: Re-enable the diagnostics egress probes on the gateway container and collect the snapshot again, or run the three tests by hand from the appliance host. The probes are the only outbound requests the appliance makes on its own. Verify: diagnostics · appliance.egress |
| check 2 fails | dns-broken (critical) | The appliance resolves with this network's resolver. A resolver that cannot answer for the provider or channel hosts fails every outbound call identically, and a browser on a laptop keeps working because it resolves over DNS-over-HTTPS. Nothing downstream of resolution can be diagnosed until this is fixed. Fix: Set a resolver that answers for the hosts in the DNS section of the snapshot, then collect the snapshot again. See Network prerequisites. Verify: diagnostics · appliance.dns |
| check 3 fails | proxy-not-configured (critical) | This site requires an outbound proxy and the appliance has none, so every outbound connection is attempted directly and times out. A timeout reads as a dead network rather than as a missing setting, which is why the network is reported as open. Fix: Configure the outbound proxy on the appliance and restart the gateway container so it re-reads the setting. Docker itself reads neither the shell environment nor apt's configuration, so a proxy set for the shell alone does not reach image pulls. Verify: diagnostics · appliance.proxy.configured = true |
| check 4 fails | https-path-blocked (critical) | Names resolve and the HTTPS path does not complete. DNS, a TCP connection on 443, and an authenticated HTTPS request fail independently: a firewall that permits ICMP and blocks 443 is ordinary, and Ubuntu's mirrors are served over port 80, so ping and apt both succeed while this fails. The three tests are not interchangeable. Fix: Allow direct outbound HTTPS from the appliance to the hosts the snapshot names as failing, or route them through the site proxy and configure it on the appliance. Read each failing target's detail line in the snapshot before changing a rule — a 407 is a proxy credential problem, a timeout is a blocked path, and a TLS error is inspection. Verify: diagnostics · appliance.update.channelReachable = true |
| check 5 fails | tls-inspection-untrusted (critical) | A middlebox terminates TLS and presents a certificate the appliance does not trust, so the connection fails at the handshake. The detail line reports a TLS error rather than a timeout, which is the only way to tell this apart from a blocked port. Fix: Install your site's CA certificate on the appliance so it trusts the inspection device, or exempt the appliance's outbound destinations from inspection. Both are legitimate; pick the one your security team already operates. Verify: expected · The failing egress targets report ok, or the issuer is now trusted by the appliance. |
Note
A TLS-inspecting middlebox produces a TLS error, not a timeout. Read each failing target's detail line in the diagnostic snapshot before changing a firewall rule: a 407 is a proxy credential problem, a timeout is a blocked path, and a handshake error is inspection.
Limits
Rate limits and quotas
A gateway throttle, an API-key quota and the provider's own 429 look alike to the caller and have completely different remedies. Only the first two are settings on this appliance.
A request is refused for being too frequent, or a quota is exhausted
You might see: Too many requests, please try again later. · Too many help requests, please try again in a few minutes. · Too many unlock requests, please try again later. · Too many attestation requests
Checks
- 1
Establish which limit refused the call — the gateway's own throttle, an API-key quota, or the provider's
expected · You can say which of the three it is.
- 2
For a quota message, read the key's configured request and token quotas
expected · The key has headroom left in the current window.
- 3
Check that Redis is reachable, because the throttles keep their counters there
diagnostics · datastores.redis.reachable = true
Causes and fixes
| Reached when | Cause | Explanation, fix and verification |
|---|---|---|
| check 1 fails | cannot-attribute (info) | A gateway throttle and a provider refusal look alike to the caller and have completely different fixes: one is a setting on this appliance, the other is an account limit at the provider that no CID setting can raise. Fix: Find the request in Event Logs and read its route and status. A gateway throttle refuses before any outbound call, so no provider call is logged for it. Verify: expected · The route tells you which limiter fired. |
| check 2 fails | key-quota-exhausted (warning) | Gateway API keys carry optional request and token quotas, and the key is refused once either is used up. This is a deliberate cap on the key, not a fault. Fix: Raise the quota on the key, issue a second key for the workload, or wait for the window to roll. See Access keys. Verify: expected · The request is accepted. |
| check 3 fails | redis-unreachable (warning) | Redis holds the throttle counters and the scheduled work. When it is unreachable, rate limiting degrades open rather than closed, so a burst of refusals while Redis is down comes from somewhere else. Fix: Follow PostgreSQL or Redis is unreachable. Verify: diagnostics · datastores.redis.reachable = true |
A gateway API key is refused
Keys are hashed, so a lost key cannot be recovered — only replaced. The interesting case is a group key: it borrows the identity of the group's earliest-added member, so emptying the group stops the key resolving and removing that member transfers the key's identity rather than revoking it.
A gateway API key is refused
You might see: Invalid API key · Invalid or expired API key · API key is not active · API key has expired
Before you start
Checks
- 1
Find the key in the estate
expected · The key exists.
- 2
Read the key's status and expiry
expected · The key is active and either has no expiry or expires in the future.
- 3
Check what identity the key resolves to
expected · The key is assigned to a tenant, or to a group that has at least one member.
Causes and fixes
| Reached when | Cause | Explanation, fix and verification |
|---|---|---|
| check 1 fails | key-not-present (warning) | Keys are stored as an unsalted SHA-256 hash of 32 random bytes, so a key that was deleted or never created cannot be recognised and cannot be recovered from the database. Fix: Issue a new key under Access Keys and reconfigure the client with it. Verify: expected · The call is accepted. |
| check 2 fails | key-inactive-or-expired (warning) | A key carries an optional expires_at (NULL means never) and an active flag. Both refuse the call at the guard, before any route logic runs, which is why the refusal looks identical to a typo in the key.Fix: Re-activate the key, extend its expiry, or issue a replacement and rotate the client. Verify: expected · The call is accepted. |
| check 3 fails | group-key-has-no-member (warning) | A group key does not have an identity of its own — it borrows one from the group's earliest-added member. Empty the group, or remove that member, and the key resolves to no tenant and is refused. Removing that user also transfers the key's identity rather than revoking the key. Fix: Add a member to the group, or reassign the key to a tenant directly. Treat a group key as carrying a person's identity until service identities ship. Verify: expected · The call is accepted and the resulting records are attributed to the expected tenant. If that does not apply: Include the key prefix (never the key), the assignment, and the exact 401 body. Do not paste the key into a ticket — it cannot be recovered from the hash, so a leaked key must be revoked. |
Related
- Credentials — adding and testing provider keys.
- Models — what the catalogue holds and how it is populated.
- Access keys — gateway API keys, quotas and expiry.
- Network prerequisites — what must be reachable.
Last updated on