Performance Benchmarks

LLM Proxy Results

Archestra handled 851–859 requests per second versus LiteLLM’s 172–175 on the same machine.

Proxy	Requests per Second	Median Added Latency	P95 Added Latency	P99 Added Latency
Archestra	851.5–859.5	58.71 ms	88.13 ms	99.44 ms
LiteLLM	172.3–174.5	266.18 ms	331.34 ms	822.92 ms

Each proxy: four logical CPUs, 8 GiB RAM, one worker; 50 concurrent requests with authentication and usage accounting.
Per proxy: 10,000 non-streaming requests over two repeats after warmup, zero errors. RPS shows the repeat range, not maximum capacity.
Added latency is client response time minus that request’s mock-provider duration. Archestra awaits log writes; LiteLLM writes asynchronously with default content storage.

Full methodology and evidence: pinned versions, hardware, configuration, streaming results and raw measurements.

Tool calls include authentication, access checks and call logging.

Gateway	Requests per Second	Median Added Latency	P95 Added Latency	P99 Added Latency
Archestra	611.1	79.39 ms	85.75 ms	93.33 ms

16 vCPU / ~63 GiB RAM; one worker, 50 concurrent calls, 600 measured calls and zero errors.

Added latency uses the gateway’s internal timer minus the mock MCP server’s duration. It differs from the LLM comparison’s client timer.

Both tests use local mock servers, with no real inference or provider spend.

LLM comparison: Archestra 1.5.0-rc.10 versus LiteLLM 1.106.4, using LiteLLM’s default content omission. Follow the pinned harness in clean-database sensitivity mode; select its archestra and litellm-default results.
MCP gateway: the table updates with each Archestra release. The benchmark workflow retains measurements; the benchmark README covers local setup.

Run the MCP benchmark from platform/ with Node.js, pnpm, Rust and Docker installed:

bash

pnpm install --frozen-lockfile
bash benchmarks/ci.sh --requests=300 --stream-requests=300 --repeats=2