Skip to main content

Connect a model service

The built-in model connection sends every model request through LiteLLM. LiteLLM gives Agents a stable model name while keeping the upstream address, credential, routing, budget, and request log on the server. A deployment can also install another model connection. The following settings apply to the LiteLLM connection included with AstraBox.

How an Agent reaches a model service The Agent program sends a request from its sandbox to LiteLLM. When protected credential delivery is enabled, the sandbox adds a scoped gateway key. LiteLLM resolves the Agent's route name, adds the upstream credential, and calls the configured model service. The AstraBox service also asks LiteLLM for the model names shown in the Agent editor. SANDBOX PROCESSES MODEL CALL SCOPED KEY ADDED MATCHED ROUTE /v1/models Agent program selected by the Environment sends route name + request Credential protection when enabled adds the scoped gateway key Model gateway LiteLLM · bundled or your own holds the upstream credential Model service hosted or self-hosted selected by the LiteLLM route AstraBox service model names in the Agent editor LEGEND the model gateway you configure the model request path the console's model list the request leaves AstraBox

Choose the connection that fits your deployment​

SetupChoose it whenWhat you configure
Bundled LiteLLMYou are installing AstraBox on one host or evaluating itA model-service credential and a LiteLLM route
Existing LiteLLMYour organization already operates a model gatewayIts sandbox-facing URL, a scoped inference key, and optionally a separate server-side URL
Local model through LiteLLMThe model runs on your workstation or private networkA LiteLLM route whose target is reachable from the AstraBox host

In every LiteLLM setup, an Agent selects a route name such as company-code-model. LiteLLM maps that name to the actual model service. The route can move to another upstream model without changing every Agent that uses it.

Understand one model request​

  1. The Agent program sends the request in the protocol it supports.
  2. The sandbox reaches the LiteLLM address configured for this AstraBox deployment.
  3. LiteLLM looks up the Agent's route name, adds the upstream credential, and calls the model service.
  4. LiteLLM streams the response back and records the configured usage data.

The Agent program receives the model name and gateway access needed for the request. Upstream model-service keys stay with LiteLLM.

Use the bundled gateway​

The standard AstraBox container starts LiteLLM alongside the AstraBox service. Start with a credential and model for a route in the bundled configuration:

export ANTHROPIC_API_KEY="your-anthropic-api-key"
export ANTHROPIC_MODEL="your-model-name"
scripts/compose.sh up --build -d

Open Management console → Integrated services → LiteLLM gateway to manage model routes, provider credentials, keys, budgets, spend, and request logs. The bundled gateway opens through the AstraBox administrator session instead of requiring a separate LiteLLM login.

For configuration as code, the bundled routes are defined in containers/litellm/config.yaml. Mount a replacement file at /opt/astrabox/litellm/config.yaml in the AstraBox container to supply your own routes:

model_list:
  - model_name: "company-code-model"
    litellm_params:
      model: "openai/provider-model-id"
      api_base: "https://models.example.com/v1"
      api_key: os.environ/COMPANY_MODEL_API_KEY

Set COMPANY_MODEL_API_KEY on the AstraBox service, restart the deployment, and use company-code-model in the Agent form. An explicit upstream model version keeps model changes under the route administrator's control.

Connect an existing LiteLLM gateway​

Set the URL that sandboxes can reach and a key limited to inference routes:

ASTRABOX_LITELLM_BASE_URL=https://llm-gateway.example.com
ASTRABOX_LITELLM_API_KEY=sk-scoped-inference-key

ASTRABOX_LITELLM_BASE_URL is the address used for Agent model requests. Its hostname has to resolve and be reachable from the sandbox network. Setting this address disables the bundled LiteLLM process.

Some deployments expose the gateway to sandboxes through private DNS while the AstraBox service reaches it through another service-network address. Set the second address for model search in the Agent editor:

ASTRABOX_LITELLM_SERVER_BASE_URL=http://litellm.internal

To show the external gateway's browser console under Integrated services, set its public administration URL:

ASTRABOX_LITELLM_ADMIN_URL=https://llm-admin.example.com/ui/

The team deployment overlay at containers/compose.team-gateway.yaml requires the sandbox-facing gateway to use an HTTPS fully qualified domain on port 443 and requires an explicit scoped inference key.

Connect a local model​

Expose the local model through LiteLLM, then give it a stable route. For an Ollama service reachable from the AstraBox host:

model_list:
  - model_name: "local-code-model"
    litellm_params:
      model: "ollama_chat/qwen3:32b"
      api_base: "http://host.docker.internal:11434"

The configured address is resolved from the network where LiteLLM runs. On a Linux or remote Docker host, replace host.docker.internal with an address that the AstraBox container can resolve. See Deploy AstraBox.

Choose a model and LiteLLM route compatible with the request protocol and tool calling required by the selected Agent program.

Choose the model on an Agent​

Open Management console → Agents and create or edit an Agent. Select an Environment first. The Environment determines which Agent program runs and which sandbox settings, network access, and model connection it uses. The model field then searches that Environment's model connection; select a result or enter the exact route name.

The four parts have separate responsibilities:

PartResponsibility
AgentStores the selected model or route name
EnvironmentSelects the Agent program and model connection used when AstraBox prepares the runtime
Agent programFormats the request in its native model protocol
LiteLLMMaps the route to an upstream model and applies gateway routing, budgets, logging, and failover

With the built-in LiteLLM connection, deployment settings choose the inference gateway. Connection details stored on an Environment are also used when the console queries that Environment's model list. If the gateway returns no model list, the model field still accepts an exact route name.

Changing the Agent's model or its Environment does not reconfigure an active runtime in place. AstraBox resolves the current settings the next time it first prepares or rebuilds that Session runtime.

Keep model credentials outside the Agent program​

The bundled gateway holds upstream model-service keys. With protected credential delivery enabled, the Agent program receives a placeholder gateway credential, and the sandbox service adds the real value only to matching model requests. The Agent stores no model key, and the Environment editor returns only a mask for a saved value.

See Protect credentials used by Agents for the request-matching path.

Verify the connection​

Before making an Agent available:

  • run a normal response and a tool or MCP call;
  • check the errors produced by an invalid key, rate limit, and timeout;
  • confirm usage and cost attribution in LiteLLM; and
  • verify that the Environment's network rules admit the gateway address.