Add a custom provider

This page is for the admin. It explains how to add a provider that LiteLLM has no built-in knowledge of — so the dashboard shows the provider’s real name instead of a borrowed one. Use this when a provider speaks the OpenAI wire format but is not one of LiteLLM’s built-in providers. Two live examples on this gateway: Command Code (commandcode/…) and Free Model (freemodel/…).

Without this registration, the only way to reach such a provider is the openai/ prefix. That works, but the dashboard then labels the model “OpenAI” — which is false and confusing. This page fixes the label properly.

The pieces

A custom provider needs three things. All three live on the Anchor Node, not in the dashboard:

Piece	File	What it does
Provider registration	`~/apps/litellm/config.yaml`	Teaches LiteLLM the new prefix (`commandcode/`) and which handler serves it
Handler module	`~/apps/litellm/custom_providers/<name>_handler.py`	A small Python class that forwards requests to the provider’s endpoint
API key	`~/apps/litellm/.env`	The provider key, passed into the container as an environment variable

The dashboard cannot create or edit any of these. LiteLLM only supports custom providers through the config file. Everything else — model entries, pools, teams, keys — stays in the dashboard and the database.

Step by step: adding Command Code

Write the handler

Create ~/apps/litellm/custom_providers/commandcode_handler.py:

import os
from litellm import CustomLLM
from litellm.types.utils import ModelResponse

class CommandCodeConfig(CustomLLM):
    @staticmethod
    def _call(model, messages, api_base, api_key, optional_params):
        import httpx
        # LiteLLM strips the registered "commandcode/" prefix before calling us.
        # What remains is the provider's own model id — pass it through VERBATIM.
        model_id = model
        payload = {"model": model_id, "messages": messages}
        payload.update(optional_params or {})
        r = httpx.post(
            (api_base or "https://api.commandcode.ai/provider/v1") + "/chat/completions",
            headers={"Authorization": "Bearer " + (api_key or os.getenv("COMMANDCODE_API_KEY", ""))},
            json=payload, timeout=180,
        )
        r.raise_for_status()
        d = r.json()
        return ModelResponse(**{k: d[k] for k in ("id", "object", "created", "model", "choices", "usage") if k in d})

    @staticmethod
    def completion(model, messages, api_base, custom_prompt_dict, model_response, print_verbose, encoding, api_key, logging_obj, optional_params, **kwargs):
        return CommandCodeConfig._call(model, messages, api_base, api_key, optional_params)

    @staticmethod
    async def acompletion(model, messages, api_base, custom_prompt_dict, model_response, print_verbose, encoding, api_key, logging_obj, optional_params, **kwargs):
        return CommandCodeConfig._call(model, messages, api_base, api_key, optional_params)

The method signatures must match LiteLLM’s dispatch exactly — CustomLLM dispatches these as static methods. Two mistakes that each cost a debugging session here: stripping the provider prefix a second time inside the handler (LiteLLM already removed it), and forgetting the acompletion variant (the dashboard’s “Test Connection” uses it).

Under litellm_settings: in ~/apps/litellm/config.yaml:

litellm_settings:
  custom_provider_map:
    - provider: commandcode
      api_base: https://api.commandcode.ai/provider/v1
      custom_handler: custom_providers.commandcode_handler.CommandCodeConfig
    - provider: freemodel
      api_base: https://api-t2-sg.freemodel.dev/v1
      custom_handler: custom_providers.freemodel_handler.FreeModelConfig

The custom_handler path is relative to the container’s working directory — that is why the handlers directory is mounted into the container (next step).

Mount the handlers and pass the key

In ~/apps/litellm/docker-compose.yml, under the proxy service:

    volumes:
      - ./custom_providers:/app/custom_providers:ro

and under environment::

  COMMANDCODE_API_KEY: ${COMMANDCODE_API_KEY:?set in .env}

Then add the real key to ~/apps/litellm/.env:

COMMANDCODE_API_KEY=sk-...

The :? form makes the container refuse to start without the key — the same pattern as the master key.

Recreate the proxy

cd ~/apps/litellm && docker compose up -d --force-recreate proxy

A plain restart is enough for a config change, but a new volume mount needs the recreate. Health check afterwards: curl http://127.0.0.1:4000/health/liveliness from the box.

In the dashboard: Models → + Add Model. LiteLLM Model is now commandcode/deepseek-v4.1-flash — the real provider name first. Public name as usual (see Models, pools and fallbacks).

Test, both ways

Test through the dashboard’s Test Connection button on the model’s page, and through the public name:

curl -s -X POST https://api.hjc925.cloud/v1/chat/completions \
  -H "Authorization: Bearer <your-gateway-key>" \
  -H "Content-Type: application/json" \
  -d '{"model": "deepseek-v4-pro", "messages": [{"role": "user", "content": "ok"}], "max_tokens": 5}'

Both paths must succeed. The dashboard test exercises the provider path; the public-name test exercises the router path. One green and one red means the pool is broken, not the provider.

Free Model — the same pattern, shorter

Free Model differs only in the details, which shows how reusable the pattern is:

Piece	Command Code	Free Model
Prefix	`commandcode/`	`freemodel/`
Endpoint	`https://api.commandcode.ai/provider/v1`	`https://api-t2-sg.freemodel.dev/v1`
Key env var	`COMMANDCODE_API_KEY`	`FREEMODEL_API_KEY`
Handler class	`CommandCodeConfig`	`FreeModelConfig`
Handler file	`commandcode_handler.py`	`freemodel_handler.py`

The handler body is otherwise identical — copy one, rename the class, change the endpoint and the env var name.

What breaks, and what it looks like

Symptom	Cause	Fix
Proxy will not start, log shows `KeyError: 'custom_handler'`	`custom_provider_map` entry missing the `custom_handler` line	Add the handler path, or remove the whole map entry to get the door back up
`Unmapped LLM provider` on `/chat/completions`	The prefix exists but no handler is loaded (mount missing, or typo in the handler path)	Check the compose mount and the `custom_handler` path
Dashboard Test Connection fails, public-name calls succeed	Handler stripped the model id a second time	The handler must pass the model through verbatim — LiteLLM already removed the prefix
`has no attribute 'acompletion'`	Handler class missing the async variant	Add `acompletion` with the same signature
`missing 1 required positional argument: 'self'`	Methods defined as instance methods; LiteLLM dispatches them as static	Add `@staticmethod` to `completion`, `acompletion` and the internal helper
`astreaming() missing 1 required positional argument: 'self'`	Streaming requests need the `streaming`/`astreaming` methods too — non-streaming works, streaming 500s	Add both, as `@staticmethod`, yielding `GenericStreamingChunk` items
Pool keeps sending traffic to a provider that is out of quota	Handler raised a raw `httpx` error, so the router saw no `429` status and never cooled the entry down	Map to `litellm.RateLimitError` with `headers=dict(r.headers)` — see below
`'Response' object does not support the context manager protocol`	Sync `httpx.post(...)` context manager used inside `astreaming`	Use `async with httpx.AsyncClient()` + `client.stream(...)` + `aiter_lines()` in `astreaming`

Raising errors the router can act on

A handler that works is not yet a handler that behaves in a pool. The router decides whether to cool an entry down by reading the status code off the exception it catches. A raw httpx.HTTPStatusError carries no status the router recognises, so a provider returning 429 looks to the router like an ordinary failure: the entry is never cooled down, it keeps its share of the traffic, and every request routed to it fails until a human notices. Map the response to LiteLLM’s own exception types before raising, and pass the provider’s headers through on rate limits — that is where retry-after lives, and it is what lets the router hold a quota-exhausted entry until the quota actually resets rather than retrying it every minute.

import litellm

# Turn an HTTP response into the exception type the router understands.
def _raise_mapped(r, model, provider):
    body = r.text[:500]
    if r.status_code == 429:
        # headers carry retry-after, which sets the cooldown length
        raise litellm.RateLimitError(
            message=f"{provider} returned HTTP 429: {body}",
            llm_provider=provider, model=model,
            response=r, headers=dict(r.headers))
    if r.status_code in (401, 403):
        raise litellm.AuthenticationError(
            message=f"{provider} returned HTTP {r.status_code}: {body}",
            llm_provider=provider, model=model, response=r)
    if r.status_code == 400:
        raise litellm.BadRequestError(
            message=f"{provider} returned HTTP 400: {body}",
            llm_provider=provider, model=model, response=r)
    raise litellm.APIError(
        status_code=r.status_code,
        message=f"{provider} returned HTTP {r.status_code}: {body}",
        llm_provider=provider, model=model, response=r)

Call it from every path that touches the network — completion, acompletion, streaming and astreaming — not only the one you tested by hand.

LiteLLM reads the headers off exception.headers first and exception.response.headers second, so passing headers= explicitly on RateLimitError is the reliable form. The effect is visible in Models, pools and fallbacks: with the header honoured, a member that reports a 3.6-day quota reset is held out for 3.6 days instead of being retried every 60 seconds.

Forwarding reasoning and tool calls while streaming

Two fields do not survive a naive streaming handler, and both fail quietly — the call succeeds and the content is simply missing.

Field	Where it belongs in `GenericStreamingChunk`
Tool calls	`tool_use`
Reasoning / thinking text	inside `provider_specific_fields`, e.g. `{"reasoning_content": ...}`

GenericStreamingChunk accepts exactly these keys: text, tool_use, is_finished, finish_reason, usage, index, provider_specific_fields. Anything else is dropped without a warning.

A blank reply is usually not the provider’s fault. Reasoning tokens are counted against max_tokens. A model given a small budget can spend all of it thinking and return empty content with finish_reason: "length" — which reads exactly like a broken handler. Check finish_reason before blaming the integration.

Where things live afterwards

Change	Where it happens
Add another model from the same provider	Dashboard — Models → Add Model, prefix `<provider>/…`
Change the provider’s key	`.env` + `docker compose restart proxy`, or Update API Key in the dashboard for DB-stored entries
Change endpoint / add a new provider	Config file + handler — this page

Start here

Access Manager

LLM gateway — For users

LLM gateway — For the admin

Public tools

Hermes

Add a custom provider

The pieces

Step by step: adding Command Code

Free Model — the same pattern, shorter

What breaks, and what it looks like

Raising errors the router can act on

Forwarding reasoning and tool calls while streaming

Where things live afterwards

​The pieces

​Step by step: adding Command Code

​Free Model — the same pattern, shorter

​What breaks, and what it looks like

​Raising errors the router can act on

​Forwarding reasoning and tool calls while streaming

​Where things live afterwards

The pieces

Step by step: adding Command Code

Free Model — the same pattern, shorter

What breaks, and what it looks like

Raising errors the router can act on

Forwarding reasoning and tool calls while streaming

Where things live afterwards