Troubleshooting

Start with these diagnostics

modelmux version
modelmux config validate
modelmux config validate --check-provider
modelmux keys --json
modelmux logs --json --limit 100
curl http://127.0.0.1:8787/metrics

Record the request ID, requested model/group, HTTP status, selected provider/key when available, and the exact time of the failure.

The configuration does not load

• Run modelmux config validate and fix the first reported error before later errors.
• Confirm indentation uses spaces and list entries start with -.
• Verify every provider_id, model_id, and key_id reference exists.
• Check that IDs are unique and enabled entries do not reference disabled or missing dependencies.
• Confirm environment variables referenced by value_env and auth_token_env are exported in the process environment.

ModelMux returns 429 or all keys are exhausted

1. Inspect active, cooldown, invalid, limited, and disabled key counts.
2. Check model-level RPM and concurrency capacity.
3. Check each key's minute, daily, token, and concurrency limits.
4. Search recent logs for upstream 429 responses and cooldown events.
5. Confirm the requested group has at least one enabled member with an eligible key.

Do not immediately raise limits above provider capacity. Upstream 429 responses indicate that the provider or credential is already enforcing a lower effective limit.

A provider returns 401 or 403

• Verify the environment variable or secret reference resolves to the intended key.
• Check provider base URL, type, and authentication mode.
• Confirm the credential is authorized for the configured upstream model.
• Run modelmux key test --id <key>.
• Replace revoked credentials, then re-enable or retest the key.

Authentication failures can mark a key invalid. A health check cannot repair a credential that remains unauthorized.

Timeouts, network errors, and provider 5xx responses

Check DNS, firewall, TLS, and the provider base URL from the same host or container running ModelMux. Compare providers[].timeout_seconds with normal model latency, especially for long streaming generations.

Repeated transient errors place keys into cooldown. If every key uses the same unavailable provider, adding more retries will not restore service and may increase latency.

SQLite is locked or cannot be opened

• Confirm only one ModelMux instance uses the database file.
• Check directory ownership and write permissions for the database, WAL, and SHM files.
• Do not place the database on an unsuitable shared network filesystem.
• Do not remove -wal or -shm files while the service is running.
• Verify the disk has available space and the filesystem is writable.

The encrypted secret store cannot be opened

• Export the same MODELMUX_MASTER_KEY used when the file was created.
• Confirm the process can read secrets.enc.
• Run modelmux secret verify.
• Restore the encrypted backup and matching master key together.
• Do not edit the encrypted file manually.

Authentication behaves unexpectedly behind a reverse proxy

Ensure the client sends Authorization: Bearer ... and the reverse proxy forwards that header. Keep both server.require_auth and server.admin.require_auth enabled where appropriate.

Do not bypass admin authentication because the reverse proxy connects from localhost or a trusted container network. Enforce access at both layers.