Adding LLM Providers

Overview

This guide covers how to add a new LLM provider to Archestra Platform. Each provider requires:

1. LLM Proxy - The proxy that sits between clients and LLM providers. Handles security policies, tool invocation controls, metrics, and observability. Clients send requests to the proxy, which forwards them to the provider. It must handle both streaming and non-streaming provider responses.

2. Chat - The built-in chat interface.

Getting Started: Let TypeScript Guide You

The fastest way to find every file that needs updating is to add your provider's ID to the SupportedProvidersSchema enum in shared/model-constants.ts and then run pnpm typecheck. TypeScript will report compile errors everywhere a Record<SupportedProvider, ...> or exhaustive switch is missing your new provider — these are exactly the files you need to update.

Use the detailed sections below for guidance on how to implement each piece.

LLM Proxy

Provider Registration

Defines the provider identity used throughout the codebase for type safety and runtime checks.

Type Definitions

Each provider needs Zod schemas defining its API contract. TypeScript types are inferred from these schemas.

Adapter Implementation

The adapter pattern provides a provider-agnostic API for business logic. LLMProxy operates entirely through adapters, never touching provider-specific types directly.

Adapters to Implement:

• RequestAdapter: Provides read/write access for the request data (model, messages, tools);
• ResponseAdapter: Provides read/write access to the response data (id, model, text, tool calls, usage);
• StreamAdapter: Process streaming chunks incrementally, accumulating data required for the LLMProxy logic;
• LLMProvider: Create adapters, extract API keys from headers, create provider SDK clients, execute requests;

Route Handler

HTTP endpoint that receives client requests and delegates to handleLLMProxy().

Important: Deterministic Codegen

Routes must always be registered regardless of whether the provider is enabled. This ensures OpenAPI schema generation is deterministic across environments.

• Register routes unconditionally (for schema generation)
• Conditionally register HTTP proxy only when provider is enabled (has baseUrl configured)
• Return a 500 error in route handlers if provider is not configured at runtime

// ✅ Correct: Routes always registered, proxy conditionally registered
if (config.llm.{provider}.enabled) {
  await fastify.register(fastifyHttpProxy, {
    upstream: config.llm.{provider}.baseUrl as string,
    prefix: API_PREFIX,
    rewritePrefix: "",
    preHandler: createProxyPreHandler({
      apiPrefix: API_PREFIX,
      endpointSuffix: CHAT_COMPLETIONS_SUFFIX,
      upstream: config.llm.{provider}.baseUrl as string,
      providerName: "{Provider}",
      rewritePrefix: "",  // match the rewritePrefix above
    }),
  });
}

// In route handlers, check at runtime:
if (!config.llm.{provider}.enabled) {
  return reply.status(500).send({
    error: { message: "{Provider} is not configured. Set ARCHESTRA_{PROVIDER}_BASE_URL to enable.", type: "api_internal_server_error" }
  });
}

Configuration

Base URL configuration allows routing to custom endpoints (e.g., Azure OpenAI, local proxies, testing mocks).

Don't forget: Document ARCHESTRA_{PROVIDER}_BASE_URL and ARCHESTRA_CHAT_{PROVIDER}_API_KEY in docs/pages/platform-deployment.md under the Environment Variables section.

Tokenizer

Note: This is a known abstraction leak that we're planning to address in future versions. Thanks for bearing with us!

Tokenizers estimate token counts for provider messages. Used by Model Optimization and Tool Results Compression.

Model Optimization

Note: This is a known abstraction leak that we're planning to address in future versions. Thanks for bearing with us!

Model optimization evaluates token counts to switch to cheaper models when possible.

Tool Results Compression

Note: This is a known abstraction leak that we're planning to address in future versions. Thanks for bearing with us!

TOON (Token-Oriented Object Notation) compression converts JSON tool results to a more token-efficient format. Each provider needs its own implementation because message structures differ.

The function must:

1. Iterate through provider-specific message array structure
2. Find tool result messages (e.g., role: "tool" in OpenAI, tool_result blocks in Anthropic, functionResponse parts in Gemini)
3. Parse JSON content and convert to TOON format using @toon-format/toon
4. Calculate token savings using the appropriate tokenizer
5. Return compressed messages and compression statistics

Dual LLM

Note: This is a known abstraction leak that we're planning to address in future versions. Thanks for bearing with us!

Dual LLM pattern uses a secondary LLM for Q&A verification of tool invocations. Each provider needs its own client implementation.

Metrics

Note: This is a known abstraction leak that we're planning to address in future versions. Thanks for bearing with us!

Prometheus metrics for request duration, token usage, and costs. Requires instrumenting provider SDK clients.

For example: OpenAI and Anthropic SDKs accept a custom fetch function, so we inject an instrumented fetch via getObservableFetch(). Gemini SDK doesn't expose fetch, so we wrap the SDK instance directly via getObservableGenAI().

Frontend: Logs UI

Interaction handlers parse stored request/response data for display in the LLM Proxy Logs UI (/logs/llm-proxy).

E2E Tests

Each provider must be added to the LLM Proxy and Chat UI e2e tests to ensure all features work correctly.

LLM Proxy E2E Tests

All LLM Proxy test files use the satisfies Record<SupportedProvider, Config> pattern to enforce at compile time that every provider has a config entry. When a new provider is added to SupportedProvider without adding its test config, TypeScript will report a compile error.

Chat Support

Below is the list of modifications required to support a new provider in the built-in Archestra Chat.

Configuration

Environment variables for API keys and base URLs.

Chat Provider Registration

Allows users to select this provider's models in the Chat UI.

Model Listing

Each provider has a different API for listing available models.

LLM Client

Chat uses Vercel AI SDK which requires provider-specific model creation.

Error Handling

Each provider SDK wraps errors differently, requiring provider-specific parsing.

Frontend UI

UI components for Chat need provider-specific configuration.

Reference Implementations

Existing provider implementations for reference:

Full implementations (custom API formats):

• OpenAI: backend/src/routes/proxy/routesv2/openai.ts, backend/src/routes/proxy/adapterV2/openai.ts
• Anthropic: backend/src/routes/proxy/routesv2/anthropic.ts, backend/src/routes/proxy/adapterV2/anthropic.ts
• Cohere: backend/src/routes/proxy/routesv2/cohere.ts, backend/src/routes/proxy/adapterV2/cohere.ts
• Gemini: backend/src/routes/proxy/routesv2/gemini.ts, backend/src/routes/proxy/adapterV2/gemini.ts
• Bedrock: backend/src/routes/proxy/routesv2/bedrock.ts, backend/src/routes/proxy/adapterV2/bedrock.ts (uses AWS Signature V4 auth and Converse API)

OpenAI-compatible implementations (reuse OpenAI types/adapters with minor modifications):

• Groq: backend/src/routes/proxy/routesv2/groq.ts, backend/src/routes/proxy/adapterV2/groq.ts (best starting point — cleanest example of OpenAI reuse)
• xAI: backend/src/routes/proxy/routesv2/xai.ts, backend/src/routes/proxy/adapterV2/xai.ts
• vLLM: backend/src/routes/proxy/routesv2/vllm.ts, backend/src/routes/proxy/adapterV2/vllm.ts
• Ollama: backend/src/routes/proxy/routesv2/ollama.ts, backend/src/routes/proxy/adapterV2/ollama.ts
• ZhipuAI: backend/src/routes/proxy/routesv2/zhipuai.ts, backend/src/routes/proxy/adapterV2/zhipuai.ts

Tip: If adding support for an OpenAI-compatible provider (e.g., Azure OpenAI, Together AI), use the Groq implementation as a starting point — it re-exports OpenAI's type definitions, message schemas, and tool schemas with minimal boilerplate.

Smoke Testing

Use PROVIDER_SMOKE_TEST.md during development to verify basic functionality. This is a quick, non-exhaustive list.

Note, that Archestra Chat uses streaming for all LLM interactions. To test non-streaming responses, use an external client like n8n Chat node.