Core concepts

Understand providers, models, keys, groups, runtime state, and how they work together.

Providers

A provider describes an upstream API: its base URL, protocol type, authentication behavior, timeout, and enabled state. Multiple models and keys can reference one provider.

Provider IDs are internal configuration identifiers. They should be stable and unique, such as openai-prod, anthropic-team, or local-vllm.

Models

A model is the public routing target used by clients. Its id is sent in the request, while model_name is forwarded to the upstream provider.

config.yaml

models:
  - id: coding-fast
    provider_id: deepseek
    model_name: deepseek-coder
    strategy: least_error
    requests_per_minute: 120
    max_concurrent_requests: 10
    enabled: true

This separation lets applications depend on stable ModelMux IDs while upstream model names or providers change behind the gateway.

API keys

A key belongs to one provider and model. It contains a credential reference plus routing and limit settings.

Field	Purpose
`priority`	Ordering for failover selection. Lower values are considered first.
`status`	Configured state, normally active or disabled.
`value_env`	Environment variable containing the provider credential.
`secret_ref`	Reference in the encrypted secret store.
`requests_per_minute`	Maximum requests in the rolling minute window.
`tokens_per_minute`	Maximum combined input and output tokens per minute.
`max_concurrent_requests`	Maximum simultaneous requests using the key.
`daily_request_limit`	Maximum requests per day.
`daily_token_limit`	Maximum total tokens per day.

Model groups

A model group is a client-visible alias that can route across several models or exact keys. Groups are useful for cross-provider failover, cost tiers, quality tiers, or workload aliases.

config.yaml

model_groups:
  - id: production-chat
    name: Production Chat
    strategy: weighted
    enabled: true
    members:
      - model_id: primary-chat
        priority: 1
        weight: 3
        enabled: true
      - model_id: backup-chat
        priority: 2
        weight: 1
        enabled: true
      - key_id: emergency-key
        priority: 3
        weight: 1
        enabled: true

A model_id member keeps the selected model's normal key-pool behavior. A key_id member pins routing to one exact key.

Runtime key state

State	Meaning
Active	Eligible when limits and routing conditions allow.
Disabled	Explicitly excluded by configuration or an admin action.
Cooldown	Temporarily excluded after a rate limit, timeout, server error, or transient failure.
Invalid	Excluded after authentication failures such as 401 or 403.
Limited	Currently unavailable because a local quota or concurrency limit is exhausted.

Background health checks can recover keys after transient failures. Invalid credentials require correcting the credential before they can become healthy.

Selection hierarchy

Resolve the requested model ID or model-group ID.
If a group was requested, select an enabled member using the group strategy.
Build the candidate key pool for the selected model, or use the exact pinned key.
Remove keys that are disabled, invalid, cooling down, or locally limited.
Choose a candidate using the model's key-rotation strategy.
Attempt the upstream request and apply retry/failover rules when necessary.