Skip to main content

Models and routing

Applications request a model group (for example gpt-4o). The gateway maps that name to one or more deployments — a provider + credential + provider-specific model id, each with a weight.

Model groups​

A model group is the public name your SDKs send in model. Several deployments can sit behind the same group (for example two Azure regions plus an OpenAI backup).

GET /v1/models lists groups the calling key is allowed to use. An empty allowlist on the key means every enabled group.

Deployments​

Each deployment has:

  • Provider (OPENAI, AZURE_OPENAI, ANTHROPIC)
  • Provider credential (AES-256-GCM encrypted, or akv: Key Vault reference)
  • Provider model id
  • Weight (used for weighted random pick)
  • Enabled / healthy flags
  • Input / output pricing (used for spend)

Configure these on the Models page (platform admin).

Routing, retries and fallbacks​

  1. Pick a healthy, enabled deployment in the group using weights
  2. Try up to three attempts within the group
  3. If a fallback chain is configured, try the next model group
  4. If every attempt fails, return 502 upstream_unavailable

When a fallback fires, the response includes x-zeallm-fallback-from (the original group) and x-zeallm-model-group (the group that served the call).

Anthropic is translated to and from the OpenAI wire format. Embeddings are OpenAI / Azure OpenAI only.

Access groups​

An access group is a curated bundle of model groups. Keys (and key requests) pick an access group so reviewers do not have to list models by hand. Edits to a group apply to new approvals; existing keys keep the snapshot they were minted with.

Default access groups and budget tiers live under Access & Tiers.