Rate limiting

Protect models and credentials with RPM, token, concurrency, and daily quota limits.

Limit levels

Level	Configuration	Scope
Model RPM	`models[].requests_per_minute`	All traffic targeting one model.
Model concurrency	`models[].max_concurrent_requests`	Simultaneous requests for one model.
Key RPM	`keys[].requests_per_minute`	Requests using one credential.
Key tokens/minute	`keys[].tokens_per_minute`	Combined input and output tokens for one key.
Key concurrency	`keys[].max_concurrent_requests`	Simultaneous requests using one key.
Daily request quota	`keys[].daily_request_limit`	Requests per key per day.
Daily token quota	`keys[].daily_token_limit`	Tokens per key per day.

Evaluation order

Model-wide RPM capacity is checked.
Model-wide concurrency capacity is checked.
The router filters key candidates using per-key minute, daily, and concurrency limits.
The routing strategy selects one of the remaining candidates.
Usage is updated from the request and provider response.

Model-level limits protect the complete route. Key-level limits protect individual provider credentials.

Configuration example

config.yaml

models:
  - id: production-chat
    provider_id: provider-a
    model_name: upstream-chat
    strategy: round_robin
    requests_per_minute: 200
    max_concurrent_requests: 20
    enabled: true

keys:
  - id: key-a
    provider_id: provider-a
    model_id: production-chat
    value_env: PROVIDER_KEY_A
    requests_per_minute: 60
    tokens_per_minute: 30000
    max_concurrent_requests: 4
    daily_request_limit: 5000
    daily_token_limit: 2000000

When capacity is exhausted

An exhausted key is skipped while another eligible key exists. When the complete model or group has no usable capacity, ModelMux returns HTTP 429.

A local 429 can therefore mean either the model-wide limit is full or every candidate key is unavailable, limited, disabled, invalid, or cooling down. Inspect key state and recent logs to identify the exact cause.

Restart semantics

Minute windows are maintained in memory and reset when the process restarts. Daily counters, key runtime state, quotas, and cooldowns can survive restarts when SQLite storage is enabled.

Do not use process restarts as a quota-reset mechanism. Provider-side limits remain in effect regardless of local state.

Capacity planning

Set local limits below provider-advertised limits when upstream enforcement is strict or bursty.
Reserve concurrency for health checks and operational tests.
Use token limits for workloads with highly variable prompt and response sizes.
Use model-wide limits to prevent a large key pool from exceeding application or infrastructure capacity.
Monitor cooldown frequency. Repeated cooldowns usually indicate local limits are too high or upstream capacity is insufficient.