AdminTracing

Tracing

Send model requests, tool calls, and backend logs to your OpenTelemetry collector.

Configuration

  1. Set ARCHESTRA_OTEL_EXPORTER_OTLP_ENDPOINT to the collector's OTLP HTTP base URL, usually port 4318:

    bash
    ARCHESTRA_OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
    
  2. If required, set ARCHESTRA_OTEL_EXPORTER_OTLP_AUTH_BEARER, or both ARCHESTRA_OTEL_EXPORTER_OTLP_AUTH_USERNAME and ARCHESTRA_OTEL_EXPORTER_OTLP_AUTH_PASSWORD. Bearer authentication takes precedence.

  3. Set ARCHESTRA_OTEL_TRACES_SAMPLE_RATE if you need to reduce ingest volume.

  4. Restart the backend and send a chat message.

  5. Search your tracing backend for service Archestra and a span named chat <agent name>.

Archestra appends /v1/traces and /v1/logs to the base URL.

Collect Metrics Alongside Traces

Configure Prometheus scraping alongside traces. It includes Guardrails and OpenAPPA hook and engine metrics without additional setup.

Logs

Archestra sends its backend logs to the same collector. Each record carries trace_id and span_id, plus session_id inside an agent session. Your tracing backend uses them to jump from a span to its logs.

Content Capture

Archestra records message and tool content as span events:

SpanEventContent
LLMgen_ai.content.promptThe request messages
LLMgen_ai.content.completionThe response text
MCPgen_ai.content.inputThe tool arguments
MCPgen_ai.content.outputThe tool result

Each event is truncated to 10,000 characters; ARCHESTRA_OTEL_CONTENT_MAX_LENGTH changes the limit.

Capture is on by default. Set ARCHESTRA_OTEL_CAPTURE_CONTENT=false to leave the events out. With content encryption on, capture is off by default. This keeps encrypted content from reaching your telemetry backend as plaintext. ARCHESTRA_OTEL_CAPTURE_CONTENT=true turns it back on.

When content capture is disabled or a session suppresses content capture, LLM span error statuses also omit provider error messages.

Verbose Tracing

By default, Archestra exports only agent spans: LLM calls, MCP tool calls, agent runs, and Knowledge Base calls. Set ARCHESTRA_OTEL_VERBOSE_TRACING=true to also export HTTP routes, outgoing HTTP calls, database queries, and DNS lookups. Use it for debugging. It multiplies span volume.

Trace Structure

A session has one stable trace identity across turns and backend replicas. Each turn, inference request, tool call, and Guardrails hook is a separate span in its waterfall. The original session ID remains available as gen_ai.conversation.id; the trace ID is a stable 128-bit identifier derived from it. Calls without a session ID keep independent traces.

The root span's route.category tells you how the agent was invoked:

Invocationroute.category
Chat UIchat
A2Aa2a
Slack, Microsoft Teams, Telegramchatops
Emailemail

Calls from your own applications to the LLM Proxy (route.category=llm-proxy) and the MCP Gateway (route.category=mcp-gateway) join the trace for their recorded session identity.

To group LLM Proxy calls into one trace, send X-Archestra-Session-Id with your conversation ID. Archestra records it as gen_ai.conversation.id. When Guardrails are active, X-Appa-Session-ID takes precedence as the recorded session identity. MCP Gateway spans use the Guardrails session identity when one is present.

Investigate a Session

Open Session Investigation in Grafana and click a session row. Session Trace shows the session’s calls and policy checks. Session Logs, below it, shows inference, tool, Guardrails, and correlated application events. Select a session by its ID; the trace lookup happens automatically.

The session list includes first and last request times, agent, user, models, credential names, request count, token usage, elapsed time, total inference time, and estimated costs split by billed and subscription-covered usage. Totals come from persisted session summaries and include requests outside the dashboard's selected time range. The selected range determines which recent summaries appear. Both Loki and Tempo are required.

Summaries contain metadata only. Prompts, responses, credential values, and policy text are not included. New summaries and session traces appear as requests run after deployment; older telemetry is not rewritten.

LLM Spans

LLM spans use {operation} {model}; Knowledge Base embeddings use embedding {model}. Use these attributes to filter requests and compare token use and cost.

AttributeValue
gen_ai.operation.namechat or generate_content
gen_ai.provider.nameThe provider, for example openai or anthropic
gen_ai.request.model, gen_ai.response.modelThe requested model and the model that answered
gen_ai.request.streamingtrue or false
server.addressThe provider's base URL
gen_ai.response.id, gen_ai.response.finish_reasonsThe provider's response ID and stop reasons, such as ["stop"] or ["tool_calls"]
gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.usage.total_tokensToken counts. Input includes cached tokens.
gen_ai.usage.cache_read.input_tokens, gen_ai.usage.cache_creation.input_tokensPrompt-cache reads and writes, part of the input tokens
archestra.usage.cache_creation.1h_input_tokensCache writes at the one-hour TTL (Anthropic and Bedrock)
gen_ai.usage.reasoning.output_tokensReasoning tokens, part of the output tokens (OpenAI and Gemini)
archestra.costEstimated cost in USD. Needs model pricing.
archestra.billing.modemetered or subscription
archestra.trigger.sourceWhere the call came from, for example api, chat, chatops:slack, or knowledge:embedding
archestra.auth.methodHow the request authenticated, for example virtual_key or oauth_user
archestra.virtual_key.id, archestra.passthrough_virtual_key.idThe virtual key used, if any
archestra.app.id, archestra.app.nameThe app that made the call, if any
archestra.external_agent_idThe X-Archestra-Agent-Id header value
archestra.run.idThe X-Archestra-Run-Id header value
error.typeThe error class, when the call fails

gen_ai.usage.input_tokens includes cached tokens. The llm_tokens_total{type="input"} metric does not. It reads lower when prompt caching is active.

Knowledge Base embedding and reranking calls have no agent attributes. Their archestra.trigger.source is knowledge:embedding or knowledge:reranker.

MCP Tool Call Spans

Each tool call through the MCP Gateway is a span named execute_tool {tool_name}, for example execute_tool github__list_repos.

AttributeValue
gen_ai.operation.nameexecute_tool
gen_ai.tool.nameThe full tool name
gen_ai.tool.call.idThe tool call ID
mcp.server.nameThe MCP server, for example github
mcp.is_error_resulttrue when the tool returned an error result
mcp.blockedtrue when a guardrail blocked the call
mcp.blocked_reasonWhy the call was blocked
error.typeThe error class, when the call throws

A blocked call never runs. Its span records the policy decision with span status ERROR and the reason as the status message.

Identity Attributes

LLM spans, MCP spans, and agent run root spans carry who made the request:

AttributeValue
gen_ai.agent.id, gen_ai.agent.nameThe Archestra agent. LLM Proxy calls report LLM Proxy.
archestra.agent.typeagent, llm_proxy, mcp_gateway, or profile
archestra.agent.label.<key>Each agent label, for example archestra.agent.label.environment=production
archestra.agent.team.ids, archestra.agent.team.namesThe agent's teams (arrays)
archestra.agent.team.label.<key>The agent's team labels, merged per key across its teams (arrays)
archestra.user.id, archestra.user.email, archestra.user.nameThe user who made the request, when known
archestra.user.team.ids, archestra.user.team.namesThe user's teams (arrays)
archestra.user.team.label.<key>The user's team labels, merged per key (arrays)
gen_ai.conversation.idThe session ID

The LLM Proxy has no labels or teams. Filter its traffic by the archestra.user.team.*, archestra.external_agent_id, or archestra.app.name attributes.

Proxy Request Log Events

Attribute logical LLM Proxy and unified model API outcomes—including throttled, denied, cancelled, and midstream-failed requests—to a model, agent, user, app, or virtual key with the archestra.llm_proxy.request OpenTelemetry log event.

When an inference request handled by the shared proxy or unified model API finishes, logger archestra-llm-proxy emits one structured log record with body "LLM proxy request completed" (INFO on success or an incomplete terminal, WARN on failure) attached to the active trace context (trace_id and span_id). Like llm_proxy_request_duration_seconds, it records one event per logical request across internal retries and failovers. Model listing, unsupported passthrough endpoints, and provider file routes have no logical request event.

Authentication, authorization, and validation failures that reach Archestra but are rejected before the proxy handler emit archestra.llm_proxy.rejected with WARN severity. Search logger archestra-llm-proxy for that event name. Its attributes include archestra.proxy.denial_kind (authentication, authorization, or validation), archestra.proxy.request_id, http.request.method, http.route, http.response.status_code, and client.address. The application log carries the same fields and message LLM proxy request rejected before handler. These records contain no credentials, request body, query string, or path parameters. Credential failures detected inside the inference handler instead appear as archestra.llm_proxy.request with a 4xx http.response.status_code; search both event names for failed credentials. These events are not persisted audit events and do not appear in GET /api/audit-logs. Use ingress access logs for traffic rejected before it reaches Archestra.

Emission is best-effort and unsampled, so every completed proxy inference request emits archestra.llm_proxy.request regardless of ARCHESTRA_OTEL_TRACES_SAMPLE_RATE. The record uses a metadata-only allowlist—never prompts, completions, request or response bodies, headers, or error messages—and is not durable log replication. Use LLM spans for token and cost attribution, limit metrics for rule evaluations, and audit logs for administrative actions.

AttributeValue
gen_ai.provider.nameProvider name (for example openai or anthropic), or router if a unified-model request fails before provider selection
gen_ai.operation.nameInference operation, such as chat, generate_content, or embedding
gen_ai.request.streamingtrue or false
http.response.status_codeCommitted HTTP status code (200 when a stream fails after headers are sent, 499 on client disconnect before headers)
archestra.proxy.outcomesuccess, incomplete, error, throttled, denied, cancelled, timeout, or overloaded
archestra.proxy.error_originnone for success or incomplete, archestra, provider, or client
archestra.proxy.duration_secondsEnd-to-end logical request duration in seconds, including preflight and streaming
error.typeNormalized failure status code as a string (such as "429" or "502"), present on failed requests even when http.response.status_code is 200
gen_ai.request.modelRequested model name, when resolved
archestra.organization.id, gen_ai.agent.id, archestra.user.idOrganization, agent, and user IDs, when available
archestra.virtual_key.id, archestra.passthrough_virtual_key.idVirtual key or passthrough virtual key ID, when used
archestra.app.idApp ID, when called from an app
archestra.trigger.sourceInvocation source, such as api, chat, or email, when available
archestra.model_pool.member_model, archestra.model_pool.id, archestra.provider_api_key.id, archestra.proxy.pool_attemptsConfigured final model-pool member model, pool ID, provider credential ID, and number of pool attempts, when a pool is used