Routing and failover

Choose keys and group members with failover, round-robin, least-error, least-used, and weighted strategies.

Key-rotation strategies

Strategy	Selection behavior	Good fit
`failover`	Uses the lowest-priority eligible key first.	Primary/backup credentials.
`round_robin`	Distributes requests across eligible keys in sequence.	Even traffic distribution.
`least_error`	Chooses the eligible key with the fewest recorded errors.	Pools with uneven reliability.
`least_used`	Chooses the key with the lowest daily usage.	Balancing daily quotas.

The strategy is configured on the model. It operates after unavailable keys have been filtered out.

Model-group strategies

Strategy	Behavior
`failover`	Selects members by priority, using later members when earlier members are unavailable.
`round_robin`	Cycles across enabled group members.
`weighted`	Distributes selection according to each member's weight.

Group selection happens before normal per-model key selection. A member referencing key_id bypasses the model pool and selects that exact key.

Key eligibility

A configured key is eligible only when all relevant conditions pass:

The key, provider, and model are enabled.
The key is not explicitly disabled or marked invalid.
No active cooldown is blocking it.
Per-minute request and token limits have capacity.
Daily request and token quotas have capacity.
The concurrency limit has an available slot.

When all keys are ineligible, ModelMux returns 429 rather than forwarding a request that cannot be served safely.

Retry behavior

config.yaml

retry:
  max_retry_per_key: 1
  max_total_attempts: 5
  backoff_milliseconds: [300, 700, 1500]

max_retry_per_key limits repeated attempts against the same credential. max_total_attempts caps attempts across the complete route. Backoff values reduce immediate retry pressure during transient incidents.

Avoid retry storms

Application-level retries and ModelMux retries can multiply each other. Keep both layers bounded and use jitter or backoff in the client.

Cooldown behavior

config.yaml

cooldown:
  rate_limit_seconds: 300
  server_error_seconds: 60
  timeout_seconds: 60

Rate-limit responses typically need longer cooldowns than isolated network or server failures. During cooldown, the key remains configured but is removed from candidate selection.

When health checks are enabled, recovered keys can return to active routing automatically.

Common routing patterns

Primary and backupUse failover with priorities 1, 2, and 3.Shared team keysUse round-robin plus per-key RPM and concurrency caps.Cross-provider aliasUse a group whose members reference several models.Quota balancingUse least-used with daily request or token limits.