modelmux
quick_start →
Open documentation index
Getting startedOverviewInstallationQuick startCore concepts
ConfigurationConfiguration referenceProvidersRouting and failoverRate limiting
OperationsStorageObservabilitySecurityTUI dashboardAI routing
ReferenceCLI referenceAdmin APITroubleshooting
Docs/Getting started/ModelMux documentation
Getting started

ModelMux documentation

Run one OpenAI-compatible endpoint in front of multiple LLM providers and API keys, with routing, failover, limits, logs, and terminal-first operations.

ModelMux in one sentence

ModelMux is a lightweight reverse proxy that accepts OpenAI-compatible requests and routes them across configured providers, models, groups, and API keys.

What is ModelMux?

ModelMux sits between your application and upstream LLM APIs. Your application sends requests to one local or self-hosted endpoint. ModelMux then selects a model or group, chooses an eligible key, enforces limits, retries recoverable failures, and forwards the provider response back to the client.

It runs as a single Go binary. Redis, message queues, Postgres, and an external control plane are not required. Runtime state can remain in memory for local use or be persisted with embedded SQLite.

Compatible endpointKeep OpenAI-compatible clients and SDKs.
Key resilienceRotate keys and cool down unhealthy credentials.
Provider abstractionRoute to OpenAI-compatible, Anthropic, Gemini, and custom upstreams.
Operational visibilityInspect logs, metrics, key state, route traces, and the TUI.

Request flow

Client→Model or group→Eligible key→Provider→Response
  1. The client sends a request to /v1/chat/completions or another supported OpenAI-compatible route.
  2. ModelMux resolves the requested model ID. A model group may select one member before key selection begins.
  3. Disabled, invalid, cooling-down, quota-exhausted, or concurrency-limited keys are excluded.
  4. The configured routing strategy selects an eligible key.
  5. ModelMux applies provider conversion when Anthropic or Gemini is used.
  6. Recoverable failures may trigger retry and failover according to the retry and cooldown configuration.
  7. The response is streamed or returned normally, while usage, latency, status, and request metadata are recorded.

When to use ModelMux

SituationHow ModelMux helps
Multiple keys for one modelDistributes or fails over traffic across credentials.
Provider rate limitsEnforces local limits and temporarily cools down keys after upstream 429 responses.
Multiple LLM vendorsPresents one OpenAI-compatible endpoint while adapting supported provider formats.
Self-hosted applicationsKeeps routing and credentials under your control without a hosted gateway dependency.
Terminal operationsProvides CLI commands and a TUI for configuration, logs, metrics, chat, and key health.

Documentation map

Quick startInstall ModelMux and send the first proxied request.ConfigurationUnderstand every major YAML section and environment variable.Routing and failoverConfigure key strategies, groups, retries, and cooldowns.SecurityProtect proxy traffic, admin endpoints, and stored secrets.ObservabilityUse metrics, logs, request IDs, and route traces.TroubleshootingDiagnose exhausted keys, auth errors, storage locks, and bad config.

Production checklist

  • Bind to localhost unless remote access is required.
  • Enable proxy authentication before exposing the service over a network.
  • Keep server.admin.require_auth enabled.
  • Use environment variables or the encrypted secret store for provider credentials.
  • Enable SQLite when quotas, cooldown state, and request history must survive restarts.
  • Set realistic model and key-level RPM, token, and concurrency limits.
  • Monitor /metrics, request errors, cooldown counts, and active-key capacity.
  • Back up both the SQLite database and secrets.enc when encrypted secrets are used.