AdminTracing

You are viewing the docs for Archestra v1.4. View the latest docs

Tracing

Archestra exports OpenTelemetry traces and backend logs to any OTLP-compatible backend, such as Tempo, Jaeger, Honeycomb, or Grafana Cloud. Each LLM call and MCP tool call is a span that carries the agent, user, teams, model, token counts, and cost. Span names and gen_ai.* attributes follow the OpenTelemetry GenAI semantic conventions. Attributes the conventions do not cover use the archestra.* prefix.

Configuration

Set ARCHESTRA_OTEL_EXPORTER_OTLP_ENDPOINT to your collector's base URL. Archestra exports OTLP over HTTP (port 4318 by convention), not gRPC (4317).

bash
ARCHESTRA_OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318

The default is http://localhost:4318. Archestra appends /v1/traces for traces and /v1/logs for logs, and reports the service name Archestra.

The bearer token wins when both are set. Basic authentication needs both the username and the password.

Restart the backend after you change these variables. To check the export, send a chat message, then search your tracing backend for service Archestra and a span named chat <agent name>.

Logs

Archestra sends its backend logs to the same collector. Each record carries trace_id and span_id, plus session_id inside an agent session. Your tracing backend uses them to jump from a span to its logs.

Content Capture

Archestra records message and tool content as span events:

SpanEventContent
LLMgen_ai.content.promptThe request messages
LLMgen_ai.content.completionThe response text
MCPgen_ai.content.inputThe tool arguments
MCPgen_ai.content.outputThe tool result

Each event is truncated to 10,000 characters; ARCHESTRA_OTEL_CONTENT_MAX_LENGTH changes the limit.

Capture is on by default. Set ARCHESTRA_OTEL_CAPTURE_CONTENT=false to leave the events out. With content encryption on, capture is off by default. This keeps encrypted content from reaching your telemetry backend as plaintext. ARCHESTRA_OTEL_CAPTURE_CONTENT=true turns it back on.

Verbose Tracing

By default, Archestra exports only agent spans: LLM calls, MCP tool calls, agent runs, and Knowledge Base calls. Set ARCHESTRA_OTEL_VERBOSE_TRACING=true to also export HTTP routes, outgoing HTTP calls, database queries, and DNS lookups. Use it for debugging. It multiplies span volume.

Trace Structure

A session has one stable trace identity across turns and backend replicas. Each turn, inference request, tool call, and Guardrails hook is a separate span in its waterfall. The original session ID remains available as gen_ai.conversation.id; the trace ID is a stable 128-bit identifier derived from it. Calls without a session ID keep independent traces.

The root span's route.category tells you how the agent was invoked:

Invocationroute.category
Chat UIchat
A2Aa2a
Slack, Microsoft Teams, Telegramchatops
Emailemail

Calls from your own applications to the LLM Proxy (route.category=llm-proxy) and the MCP Gateway (route.category=mcp-gateway) join the trace for their recorded session identity.

To group LLM Proxy calls into one trace, send X-Archestra-Session-Id with your conversation ID. Archestra records it as gen_ai.conversation.id. When Guardrails are active, X-Appa-Session-ID takes precedence as the recorded session identity. MCP Gateway spans use the Guardrails session identity when one is present.

Investigate a Session

Open Session Investigation in Grafana and click a session row. Session Trace shows the session’s calls and policy checks. Session Logs, below it, shows inference, tool, Guardrails, and correlated application events. Select a session by its ID; the trace lookup happens automatically.

The session list includes first and last request times, agent, user, models, credential names, request count, token usage, elapsed time, total inference time, and estimated costs split by billed and subscription-covered usage. Totals come from persisted session summaries and include requests outside the dashboard's selected time range. The selected range determines which recent summaries appear. Both Loki and Tempo are required.

Summaries contain metadata only. Prompts, responses, credential values, and policy text are not included. New summaries and session traces appear as requests run after deployment; older telemetry is not rewritten.

LLM Spans

Each LLM call is a SpanKind.CLIENT span named {operation} {model}, for example chat gpt-4o-mini or generate_content gemini-2.5-flash. Knowledge Base embedding calls are named embedding {model}.

AttributeValue
gen_ai.operation.namechat or generate_content
gen_ai.provider.nameThe provider, for example openai or anthropic
gen_ai.request.model, gen_ai.response.modelThe requested model and the model that answered
gen_ai.request.streamingtrue or false
server.addressThe provider's base URL
gen_ai.response.id, gen_ai.response.finish_reasonsThe provider's response ID and stop reasons, such as ["stop"] or ["tool_calls"]
gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.usage.total_tokensToken counts. Input includes cached tokens.
gen_ai.usage.cache_read.input_tokens, gen_ai.usage.cache_creation.input_tokensPrompt-cache reads and writes, part of the input tokens
archestra.usage.cache_creation.1h_input_tokensCache writes at the one-hour TTL (Anthropic and Bedrock)
gen_ai.usage.reasoning.output_tokensReasoning tokens, part of the output tokens (OpenAI and Gemini)
archestra.costEstimated cost in USD. Needs model pricing.
archestra.billing.modemetered or subscription
archestra.trigger.sourceWhere the call came from, for example api, chat, chatops:slack, or knowledge:embedding
archestra.auth.methodHow the request authenticated, for example virtual_key or oauth_user
archestra.virtual_key.id, archestra.passthrough_virtual_key.idThe virtual key used, if any
archestra.app.id, archestra.app.nameThe app that made the call, if any
archestra.external_agent_idThe X-Archestra-Agent-Id header value
archestra.run.idThe X-Archestra-Run-Id header value
error.typeThe error class, when the call fails

gen_ai.usage.input_tokens includes cached tokens. The llm_tokens_total{type="input"} metric does not. It reads lower when prompt caching is active.

Knowledge Base embedding and reranking calls have no agent attributes. Their archestra.trigger.source is knowledge:embedding or knowledge:reranker.

MCP Tool Call Spans

Each tool call through the MCP Gateway is a span named execute_tool {tool_name}, for example execute_tool github__list_repos.

AttributeValue
gen_ai.operation.nameexecute_tool
gen_ai.tool.nameThe full tool name
gen_ai.tool.call.idThe tool call ID
mcp.server.nameThe MCP server, for example github
mcp.is_error_resulttrue when the tool returned an error result
mcp.blockedtrue when a guardrail blocked the call
mcp.blocked_reasonWhy the call was blocked
error.typeThe error class, when the call throws

A blocked call never runs. Its span records the policy decision with span status ERROR and the reason as the status message.

Identity Attributes

LLM spans, MCP spans, and agent run root spans carry who made the request:

AttributeValue
gen_ai.agent.id, gen_ai.agent.nameThe Archestra agent. LLM Proxy calls report LLM Proxy.
archestra.agent.typeagent, llm_proxy, mcp_gateway, or profile
archestra.agent.label.<key>Each agent label, for example archestra.agent.label.environment=production
archestra.agent.team.ids, archestra.agent.team.namesThe agent's teams (arrays)
archestra.agent.team.label.<key>The agent's team labels, merged per key across its teams (arrays)
archestra.user.id, archestra.user.email, archestra.user.nameThe user who made the request, when known
archestra.user.team.ids, archestra.user.team.namesThe user's teams (arrays)
archestra.user.team.label.<key>The user's team labels, merged per key (arrays)
gen_ai.conversation.idThe session ID

The LLM Proxy has no labels or teams. Filter its traffic by the archestra.user.team.*, archestra.external_agent_id, or archestra.app.name attributes.

Proxy Rejection Log Events

Authentication, authorization, and validation failures rejected before an LLM Proxy route handler emit archestra.llm_proxy.rejected with WARN severity. Search logger archestra-llm-proxy for this event. The backend application log carries the same fields and message LLM proxy request rejected before handler.

The event includes archestra.proxy.denial_kind (authentication, authorization, or validation), archestra.proxy.request_id, http.request.method, http.route, http.response.status_code, and client.address. It excludes credentials, request bodies, query strings, and path parameters. Export is best-effort and independent of trace sampling.

These events do not appear in the Audit table. Credential checks inside an inference handler are outside this event's scope; consult backend application logs for those failures. Use ingress access logs for traffic blocked before reaching Archestra.