Adding LLM Providers

Overview

This guide covers how to add a new LLM provider to Archestra Platform. Each provider requires:

1. LLM Proxy - The proxy that sits between clients and LLM providers. Handles security policies, tool invocation controls, metrics, and observability. Clients send requests to the proxy, which forwards them to the provider. It must handle both streaming and non-streaming provider responses.

2. Chat - The built-in chat interface.

Getting Started: Let TypeScript Guide You

The fastest way to find every file that needs updating is to add your provider's ID to the SupportedProvidersSchema enum in shared/model-constants.ts and then run pnpm typecheck. TypeScript will report compile errors everywhere a Record<SupportedProvider, ...> or exhaustive switch is missing your new provider — these are exactly the files you need to update.

Use the detailed sections below for guidance on how to implement each piece.

LLM Proxy

Provider Registration

Defines the provider identity used throughout the codebase for type safety and runtime checks.

Type Definitions

Each provider needs Zod schemas defining its API contract. TypeScript types are inferred from these schemas.

Adapter Implementation

The adapter pattern provides a provider-agnostic API for business logic. LLMProxy operates entirely through adapters, never touching provider-specific types directly.

Adapters to Implement:

• RequestAdapter: Provides read/write access for the request data (model, messages, tools);
• ResponseAdapter: Provides read/write access to the response data (id, model, text, tool calls, usage);
• StreamAdapter: Process streaming chunks incrementally, accumulating data required for the LLMProxy logic;
• LLMProvider: Create adapters, extract API keys from headers, create provider SDK clients, execute requests;

Route Handler

HTTP endpoint that receives client requests and delegates to handleLLMProxy().

Important: Deterministic Codegen

Routes must always be registered regardless of whether the provider is enabled. This ensures OpenAPI schema generation is deterministic across environments.

• Register routes unconditionally (for schema generation)
• Conditionally register HTTP proxy only when provider is enabled (has baseUrl configured)
• Return a 500 error in route handlers if provider is not configured at runtime

// ✅ Correct: Routes always registered, proxy conditionally registered
if (config.llm.{provider}.enabled) {
  await fastify.register(fastifyHttpProxy, {
    upstream: config.llm.{provider}.baseUrl as string,
    prefix: API_PREFIX,
    rewritePrefix: "",
    preHandler: createProxyPreHandler({
      apiPrefix: API_PREFIX,
      endpointSuffix: CHAT_COMPLETIONS_SUFFIX,
      upstream: config.llm.{provider}.baseUrl as string,
      providerName: "{Provider}",
      rewritePrefix: "",  // match the rewritePrefix above
    }),
  });
}

// In route handlers, check at runtime:
if (!config.llm.{provider}.enabled) {
  return reply.status(500).send({
    error: { message: "{Provider} is not configured. Set ARCHESTRA_{PROVIDER}_BASE_URL to enable.", type: "api_internal_server_error" }
  });
}

Configuration

Base URL configuration allows routing to custom endpoints (e.g., Azure OpenAI, local proxies, testing mocks).

Don't forget: Document ARCHESTRA_{PROVIDER}_BASE_URL and ARCHESTRA_CHAT_{PROVIDER}_API_KEY in docs/pages/platform-deployment.md under the Environment Variables section.

Tokenizer

Note: This is a known abstraction leak that we're planning to address in future versions. Thanks for bearing with us!

Tokenizers estimate token counts for provider messages. Used by Model Optimization and Tool Results Compression.

Model Optimization

Note: This is a known abstraction leak that we're planning to address in future versions. Thanks for bearing with us!

Model optimization evaluates token counts to switch to cheaper models when possible.

Tool Results Compression

Note: This is a known abstraction leak that we're planning to address in future versions. Thanks for bearing with us!

TOON (Token-Oriented Object Notation) compression converts JSON tool results to a more token-efficient format. Each provider needs its own implementation because message structures differ.

The function must:

1. Iterate through provider-specific message array structure
2. Find tool result messages (e.g., role: "tool" in OpenAI, tool_result blocks in Anthropic, functionResponse parts in Gemini)
3. Parse JSON content and convert to TOON format using @toon-format/toon
4. Calculate token savings using the appropriate tokenizer
5. Return compressed messages and compression statistics

Metrics

Note: This is a known abstraction leak that we're planning to address in future versions. Thanks for bearing with us!

Prometheus metrics for request duration, token usage, and costs. Requires instrumenting provider SDK clients.

For example: OpenAI and Anthropic SDKs accept a custom fetch function, so we inject an instrumented fetch via getObservableFetch(). Gemini SDK doesn't expose fetch, so we wrap the SDK instance directly via getObservableGenAI().

Frontend: Logs UI

Interaction handlers parse stored request/response data for display in the LLM Proxy Logs UI (/llm/logs).

Testing

Provider onboarding uses backend Vitest for proxy behavior coverage and e2e tests for the current running-stack checks.

Backend Proxy Matrix

The backend matrix covers:

• declared tool persistence
• execution ID persistence
• streaming tool call handling
• model optimization
• token cost limit enforcement
• TOON compression

The preferred test seam is the provider client created by adapterFactory.createClient(). Return a fake SDK-shaped client from the test and let the real route, handler, policy, persistence, and metrics code run around it.

Proxy E2E Tests

The current proxy e2e files are:

For built-in chat coverage, add provider entries to e2e-tests/tests/chat.spec.ts.

Chat Support

Below is the list of modifications required to support a new provider in the built-in Archestra Chat.

Configuration

Environment variables for API keys and base URLs.

Model Listing

Each provider has a different API for listing available models.

If the provider is keyless or uses cloud credentials instead of an API key, also update backend/src/services/system-key-manager.ts.

LLM Client

Chat uses Vercel AI SDK which requires provider-specific model creation.

Error Handling

Each provider SDK wraps errors differently, requiring provider-specific parsing.

Frontend UI

UI components for Chat need provider-specific configuration.

Reference Implementations

Existing provider implementations for reference:

Full implementations (custom API formats):

• OpenAI: backend/src/routes/proxy/routes/openai.ts, backend/src/routes/proxy/adapters/openai.ts
• Anthropic: backend/src/routes/proxy/routes/anthropic.ts, backend/src/routes/proxy/adapters/anthropic.ts
• Cohere: backend/src/routes/proxy/routes/cohere.ts, backend/src/routes/proxy/adapters/cohere.ts
• Gemini: backend/src/routes/proxy/routes/gemini.ts, backend/src/routes/proxy/adapters/gemini.ts
• Bedrock: backend/src/routes/proxy/routes/bedrock.ts, backend/src/routes/proxy/adapters/bedrock.ts (uses AWS Signature V4 auth and Converse API)

OpenAI-compatible implementations (reuse OpenAI types/adapters with minor modifications):

• Groq: backend/src/routes/proxy/routes/groq.ts, backend/src/routes/proxy/adapters/groq.ts (best starting point — cleanest example of OpenAI reuse)
• xAI: backend/src/routes/proxy/routes/xai.ts, backend/src/routes/proxy/adapters/xai.ts
• Azure AI Foundry: backend/src/routes/proxy/routes/azure.ts, backend/src/routes/proxy/adapters/azure.ts (reference for providers requiring custom request mutation — injects api-version query param via a custom fetch wrapper in llm-client.ts for the built-in chat feature, since the Vercel AI SDK's createOpenAI() has no defaultQuery option)
• vLLM: backend/src/routes/proxy/routes/vllm.ts, backend/src/routes/proxy/adapters/vllm.ts
• Ollama: backend/src/routes/proxy/routes/ollama.ts, backend/src/routes/proxy/adapters/ollama.ts
• ZhipuAI: backend/src/routes/proxy/routes/zhipuai.ts, backend/src/routes/proxy/adapters/zhipuai.ts

Tip: If adding support for an OpenAI-compatible provider (e.g., Together AI), use the Groq implementation as a starting point — it re-exports OpenAI's type definitions, message schemas, and tool schemas with minimal boilerplate. For providers that require custom query parameters on every request, see the Azure AI Foundry implementation for the fetchWithVersion pattern.

Smoke Testing

Use PROVIDER_SMOKE_TEST.md during development to verify basic functionality. This is a quick, non-exhaustive list.

Note, that Archestra Chat uses streaming for all LLM interactions. To test non-streaming responses, use an external client like n8n Chat node.