Hermes Provider Fallbacks

Why fallbacks matter

One failed LLM provider should not take your AI agent offline. Yet the most common failover setup — a second model behind the same router — fails at exactly the moment it is needed.

This guide comes out of a real Hermes deployment. The primary Codex route hit a 429 quota limit. The backup router then returned 503 because it was at capacity. From the operator's chat interface, the agent simply looked frozen while both routes retried.

Adding more retries would only have made the hang longer. What was needed was a fallback chain with genuinely different failure domains:

Hermes can switch to another provider:model when the primary hits rate limits, server overload, authentication failures, connection problems, or repeated invalid responses. The conversation and tool context stay intact.^[1]

The complete setup was five commands:

# Prove the primary works first
hermes model
hermes chat -q "Reply with PRIMARY_OK"

# Add backups in the order they should be tried
hermes fallback add
hermes fallback add

# Inspect the result
hermes fallback list

hermes fallback add opens the same provider and model picker used by hermes model. It appends each selection to the fallback_providers chain without replacing the primary.

Setup

1. Configure and prove the primary provider

Hermes recommends configuring the base provider before adding routing or fallback behaviour.^[2]

hermes --version
hermes model
hermes chat -q "Reply with PRIMARY_OK"

Do not add fallbacks until the primary can complete a normal request. Otherwise you cannot tell whether a later success came from the intended route or from accidental failover.

2. Add fallback providers in order

hermes fallback add
hermes fallback add
hermes fallback list

The first add becomes fallback 1, the next becomes fallback 2, and so on. The command reuses the Hermes provider/model picker and preserves the current primary model.^[1]

For the stack behind this guide, the public-safe shape is:

Primary     openai-codex / gpt-5.6-sol
Fallback 1  custom / hosted Qwen tool-capable model
Fallback 2  custom / local OmniRoute free model

The private base URLs and credentials are deliberately omitted.

3. Understand the custom-endpoint shape

For a custom OpenAI-compatible fallback, Hermes stores a real top-level YAML list. This is the canonical shape from the official documentation:^[1]

fallback_providers:
  - provider: custom
    model: hosted-qwen-tool-model
    base_url: https://llm.example.com/v1
    key_env: HOSTED_QWEN_API_KEY
  - provider: custom
    model: local-free-model
    base_url: http://127.0.0.1:20128/v1
    key_env: OMNIROUTE_API_KEY

Prefer hermes fallback add over hand-editing ~/.hermes/config.yaml. The YAML above is useful for review and managed configuration — not as a place to paste secrets.

Put secret values in the protected environment file:

chmod 600 ~/.hermes/.env

4. Verify the chain

hermes fallback list
hermes config check
hermes doctor

A healthy listing shows one primary followed by numbered fallback entries. Check that every entry has:

• a provider
• an exact model ID
• the correct base URL for custom endpoints
• credentials available through the provider auth store or a named environment variable

Start a new CLI session, or use /new on the gateway, before testing — so you are not relying on state loaded before the configuration changed.

5. Know what triggers failover

Hermes documents these primary-model triggers:^[1]

When fallback activates, Hermes resolves the backup credentials, builds the appropriate client, swaps the provider and model, resets the retry counter, and continues with the existing conversation and tool history.^[1]

Designing the chain

6. Test the failure domains, not just the models

A useful chain should differ across at least two of the following:

• provider account or subscription
• API endpoint
• cloud region or hosting platform
• authentication path
• model family
• local versus hosted infrastructure

Bad chain — both entries die when the router dies:

Primary:  router.example/v1 -> model A
Fallback: router.example/v1 -> auto

Better chain — three independent failure domains:

Primary:  Codex OAuth
Fallback: independently hosted Qwen endpoint
Fallback: local OmniRoute route

Pin exact model IDs. auto behind the same broken endpoint is not a fallback.

7. Validate tool use

A plain “hello” only proves that text generation works. Run a harmless tool-use probe against each candidate before trusting it as an agent fallback:

Use a read-only tool to report the current working directory,
then summarize the result in one sentence.

The fallback model must support the API shape, tool calls, context size, and structured outputs your workflows require. A cheap chat model that cannot call tools is not a useful fallback for an operational agent.

8. Watch the hidden cost

Provider switches reset or miss model-specific prompt caches. A long conversation may be re-read at full input-token price on the first fallback request, and again when traffic returns to the primary.^[1]

That cost is usually better than downtime, but it matters for long-running agents. Keep the fallback capable, independent, and affordable enough to absorb a full-context request.

9. Manage or remove entries

hermes fallback list
hermes fallback remove
hermes fallback clear

remove lets you choose a single entry. clear removes the entire chain after confirmation.

A CLI pitfall worth knowing

Use hermes fallback add. If configuration automation must write the YAML directly, use a structured YAML or configuration library that emits a real list — then require hermes fallback list to recognise every entry as the acceptance test.

Deployment checklist

• Primary provider works on its own
• Exact model IDs are pinned — no auto
• Fallbacks use genuinely different failure domains
• Every model passes a tool-call probe, not just a text completion
• Secrets stay out of config.yaml and out of any shared example
• hermes fallback list shows the intended order
• A fresh session loads the new chain
• Cost and context limits are acceptable on the backup routes

Sources

1. Fallback Providers — Hermes Agent documentation
2. Quickstart — Hermes Agent documentation