Adding LLM Providers

Overview

This guide covers how to add a new LLM provider to Archestra Platform. Each provider requires:

1. LLM Proxy - The proxy that sits between clients and LLM providers. Handles security policies, tool invocation controls, metrics, and observability. Clients send requests to the proxy, which forwards them to the provider. It must handle both streaming and non-streaming provider responses.

2. Chat - The built-in chat interface.

LLM Proxy

Provider Registration

Defines the provider identity used throughout the codebase for type safety and runtime checks.

Type Definitions

Each provider needs Zod schemas defining its API contract. TypeScript types are inferred from these schemas.

Adapter Implementation

The adapter pattern provides a provider-agnostic API for business logic. LLMProxy operates entirely through adapters, never touching provider-specific types directly.

Adapters to Implement:

• RequestAdapter: Provides Read/write access for the request data (model, messages, tools);
• ResponseAdapter: Provides Read/write access to thee response data (id, model, text, tool calls, usage);
• StreamAdapter: Process streaming chunks incrementally, accumulatin data required fro the LLMProxy logic;
• LLMProvider: Create adapters, extract API keys from headers, create provider SDK clients, execute requests;

Route Handler

HTTP endpoint that receives client requests and delegates to handleLLMProxy().

Important: Deterministic Codegen

Routes must always be registered regardless of whether the provider is enabled. This ensures OpenAPI schema generation is deterministic across environments.

• Register routes unconditionally (for schema generation)
• Conditionally register HTTP proxy only when provider is enabled (has baseUrl configured)
• Return a 500 error in route handlers if provider is not configured at runtime

// ✅ Correct: Routes always registered, proxy conditionally registered
if (config.llm.{provider}.enabled) {
  await fastify.register(fastifyHttpProxy, { upstream: config.llm.{provider}.baseUrl as string, ... });
}

// In route handlers, check at runtime:
if (!config.llm.{provider}.enabled) {
  return reply.status(500).send({
    error: { message: "{Provider} is not configured. Set ARCHESTRA_{PROVIDER}_BASE_URL to enable.", type: "api_internal_server_error" }
  });
}

Configuration

Base URL configuration allows routing to custom endpoints (e.g., Azure OpenAI, local proxies, testing mocks).

Feature Flags

Expose provider availability to the frontend for conditional UI rendering.

Tokenizer

Note: This is a known abstraction leak that we're planning to address in future versions. Thanks for bearing with us!

Tokenizers estimate token counts for provider messages. Used by Model Optimization and Tool Results Compression.

Model Optimization

Note: This is a known abstraction leak that we're planning to address in future versions. Thanks for bearing with us!

Model optimization evaluates token counts to switch to cheaper models when possible.

Tool Results Compression

Note: This is a known abstraction leak that we're planning to address in future versions. Thanks for bearing with us!

TOON (Token-Oriented Object Notation) compression converts JSON tool results to a more token-efficient format. Each provider needs its own implementation because message structures differ.

The function must:

1. Iterate through provider-specific message array structure
2. Find tool result messages (e.g., role: "tool" in OpenAI, tool_result blocks in Anthropic, functionResponse parts in Gemini)
3. Parse JSON content and convert to TOON format using @toon-format/toon
4. Calculate token savings using the appropriate tokenizer
5. Return compressed messages and compression statistics

Dual LLM

Note: This is a known abstraction leak that we're planning to address in future versions. Thanks for bearing with us!

Dual LLM pattern uses a secondary LLM for Q&A verification of tool invocations. Each provider needs its own client implementation.

Metrics

Note: This is a known abstraction leak that we're planning to address in future versions. Thanks for bearing with us!

Prometheus metrics for request duration, token usage, and costs. Requires instrumenting provider SDK clients.

For example: OpenAI and Anthropic SDKs accept a custom fetch function, so we inject an instrumented fetch via getObservableFetch(). Gemini SDK doesn't expose fetch, so we wrap the SDK instance directly via getObservableGenAI().

Frontend: Logs UI

Interaction handlers parse stored request/response data for display in the LLM Proxy Logs UI (/logs/llm-proxy).

E2E Tests

Each provider must be added to the LLM Proxy e2e tests to ensure all features work correctly.

Chat Support

Below is the list of modification requrest to support new Provider in the built-in Archestra Chat.

Configuration

Environment variables for API keys and base URLs.

Chat Provider Registration

Allows users to select this provider's models in the Chat UI.

Model Listing

Each provider has a different API for listing available models.

LLM Client

Chat uses Vercel AI SDK which requires provider-specific model creation.

Error Handling

Each provider SDK wraps errors differently, requiring provider-specific parsing.

Frontend UI

UI components for Chat need provider-specific configuration.

Reference Implementations

Existing provider implementations for reference:

Full implementations (custom API formats):

• OpenAI: backend/src/routes/proxy/routesv2/openai.ts, backend/src/routes/proxy/adapterV2/openai.ts
• Anthropic: backend/src/routes/proxy/routesv2/anthropic.ts, backend/src/routes/proxy/adapterV2/anthropic.ts
• Gemini: backend/src/routes/proxy/routesv2/gemini.ts, backend/src/routes/proxy/adapterV2/gemini.ts

OpenAI-compatible implementations (reuse OpenAI types/adapters with minor modifications):

• vLLM: backend/src/routes/proxy/routesv2/vllm.ts, backend/src/routes/proxy/adapterV2/vllm.ts
• Ollama: backend/src/routes/proxy/routesv2/ollama.ts, backend/src/routes/proxy/adapterV2/ollama.ts

Tip: If adding support for an OpenAI-compatible provider (e.g., Azure OpenAI, Together AI, Groq), use the vLLM/Ollama implementations as starting points - they reuse OpenAI's type definitions and adapters.

Smoke Testing

Use PROVIDER_SMOKE_TEST.md during development to verify basic functionality. This is a quick, non-exhaustive list.

Note, that Archestra Chat uses streaming for all LLM interactions. To test non-streaming responses, use an external client like n8n Chat node.