You are viewing the docs for Archestra v1.4. View the latest docs
Observability Metrics
Archestra serves Prometheus metrics in OpenMetrics format. Scrape them to chart and alert on LLM cost and latency, MCP tool calls, Agent Runtime health, and background jobs.
Scraping
Each process serves its own registry, so scrape every pod:
| Process | Endpoint | Emits |
|---|---|---|
| Web | http://<pod>:9050/metrics | Requests served by the web tier: LLM, MCP, chat, Agent Runtime, HTTP |
| Worker | http://<pod>:9000/metrics | Knowledge Base syncs and embeddings, the task queue, scheduled tasks, and the LLM and MCP calls they make |
The Helm chart runs the worker as its own Deployment by default. Sum counters across web and worker pods: an agent run from a schedule reports its LLM metrics from a worker pod.
ARCHESTRA_ changes the web port. Worker pods serve metrics on their API port, 9000.
Authentication
The endpoint is open by default. Set ARCHESTRA_ to require a bearer token on web and worker pods. Requests without it get 401.
scrape_configs:
- job_name: archestra
authorization:
type: Bearer
credentials: <ARCHESTRA_METRICS_SECRET>
kubernetes_sd_configs:
- role: pod
Helm Pod Annotations
For a Prometheus that discovers pods by annotation, add these to the chart values:
archestra:
podAnnotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9050"
prometheus.io/path: /metrics
The chart rewrites prometheus.io/port to 9000 on worker pods.
To check a pod, run curl -s http://<pod>:9050/metrics | grep "# HELP llm_". Add -H "Authorization: Bearer <secret>" when a secret is set.
Labels
Most LLM metrics share these labels:
| Label | Values |
|---|---|
provider, model | The LLM provider and model, for example anthropic and claude-sonnet-4-5 |
agent_, agent_ | The Archestra agent that made the call. All LLM Proxy traffic reports agent_name="LLM Proxy". Knowledge Base calls report agent_name="Knowledge Base" with an empty agent_. |
agent_ | agent, llm_, mcp_, or profile |
organization_ | The owning organization |
source | Where the request came from, for example api, chat, chatops:slack, email, schedule-trigger, app:llm_complete, knowledge:embedding, or knowledge:reranker |
MCP tool call metrics carry agent_, agent_, agent_, mcp_, tool_, and status (success or error).
Agent Labels
Go to an agent's Advanced tab and add a key and value under Labels. Each agent label becomes an extra label on the LLM, MCP, and agent_ metrics. Prometheus allows only letters, digits, and underscores in label names, so other characters become _: cost-center turns into cost_.
Dimensions Without Labels
Users, skills, apps, and client-supplied IDs are not metric labels. Use these sources instead:
- Users: per-user statistics, or the
archestra.user.*span attributes. - Skills and apps: per-skill and per-app statistics. App LLM spend shows in the LLM metrics as
source="app:llm_complete". - Client-supplied agent IDs from the
X-Archestra-Agent-Idheader: thearchestra.external_agent_idspan attribute. Onlyagent_carries it as a label.runs_ total - Credentials: the
archestra.virtual_key.idspan attribute.
LLM Metrics
| Metric | Type | Extra Labels | Measures |
|---|---|---|---|
llm_ | Histogram | status_ | LLM request duration |
llm_ | Counter | type (input, output) | Tokens used. input excludes prompt-cache tokens. |
llm_ | Counter | cache_ (read, write) | Prompt-cache tokens read from or written to the provider cache |
llm_ | Histogram | Input plus output tokens per request | |
llm_ | Counter | billing_, auth_ | Estimated list-price cost in USD. Needs model pricing. |
llm_ | Counter | Cost in USD of prompt-cache reads and writes | |
llm_ | Counter | USD saved by cache reads at the discounted price | |
llm_ | Histogram | Streaming: from the upstream call to the provider's first chunk | |
llm_ | Histogram | Streaming: from request receipt to the first byte sent to the client | |
llm_ | Histogram | Output token throughput | |
llm_ | Counter | Tool calls blocked by guardrails | |
agent_ | Counter | external_ | Unique runs, counted by the X-Archestra-Run-Id header |
llm_ | Gauge | window (24h, 7d) only | Distinct users with at least one LLM request in the window |
billing_ is metered for per-token billing and subscription for flat-rate subscription credentials. Billed spend is sum(llm_cost_total{billing_mode="metered"}). auth_ is provider_, virtual_, passthrough_, jwks, oauth_, oauth_, internal, or unknown.
llm_ minus llm_ is the time Archestra spends before the upstream call: authentication and guardrails.
Every replica reports the same llm_ value, so aggregate it with max(), not sum(). ARCHESTRA_ sets the refresh interval; 0 turns the metric off.
LLM and MCP metrics carry trace exemplars, so a Grafana panel can link a data point to its trace. See Exemplars.
MCP Metrics
| Metric | Type | Measures |
|---|---|---|
mcp_ | Counter | Tool calls through the MCP Gateway |
mcp_ | Histogram | Tool call duration |
mcp_ | Histogram | Tool call argument size |
mcp_ | Histogram | Tool call result size |
mcp_ | Gauge | State of each self-hosted MCP server, by server_ and state |
mcp_ is 1 for the server's current state: not_, pending, running, failed, succeeded, hibernated, or waking. count(mcp_server_deployment_status{state="running"} == 1) counts running servers.
Agent Runtime Health
| Metric | Type | Labels | Measures |
|---|---|---|---|
agent_ | Counter | Agent Runtime runs that reached a running backend | |
agent_ | Counter | outcome | Finished runs: completed, failed, stopped_, expired_, or expired_ |
agent_ | Histogram | Startup time, including scheduling and image pulls | |
agent_ | Counter | steer_ | Steering messages delivered into running sessions |
agent_ | Counter | interface, outcome | Final replies delivered to chatops or email |
agent_ | Counter | reason | Runtime tool calls Guardrails refused or held at the runtime boundary |
agent_ | Counter | outcome | Protected runtime files pinned while their runtime was mid-call (pinned), and how each pin ended (admitted or held). discard_ counts pins the backend could not release; the runtime removes those within 6 hours |
agent_ | Counter | outcome | Runtime reads a governed session made while Guardrails enforcement was off: signed so they pass once it is back on, or unsigned because the call could not be recorded |
agent_ | Gauge | agent_, backend, condition | Current task counts per condition |
agent_ | Gauge | agent_, backend, condition | Age of the oldest task per condition |
agent_ | Gauge | Time of the last health snapshot |
agent_ reasons: proof_ (a call to a runtime bound to a session carried no proof of its source), different_ (its proof names another session), unverified_, unparented, other_, and not_ (the proof or the runtime's session did not check out), client_ (an unrecognized client while those are blocked), unbindable and address_ (a launch Guardrails could not bind or refused), held (Guardrails held a value crossing out of the runtime), altered (a pinned file changed after it was admitted), and withheld (a run result not admitted to the session).
agent_ conditions are working, submitted, input_, auth_, failed_ (the last 15 minutes), and completion_. Conditions overlap: a task waiting for authentication also counts as working. agent_ conditions are heartbeat, submitted, and completion_.
Every replica reports the same health gauge values, so aggregate them with max, not sum. Alert on failed scrapes too, because missing samples do not mean healthy runs. A fresh heartbeat shows the orchestration is alive, not that the agent is making progress.
This alert finds runs whose heartbeat is more than two minutes old:
max by (agent_id, backend) (
agent_runtime_health_age_seconds{condition="heartbeat"}
) > 120
Add a pending period (for: 5m) to skip transient spikes.
Knowledge Base Metrics
| Metric | Labels | Measures |
|---|---|---|
rag_, rag_ | connector_, status | Connector syncs (success, failed, or partial) and their duration |
rag_, rag_, rag_ | connector_ | Documents read, documents added or updated, and chunks created |
rag_ | connector_ | Documents skipped because they have no extractable text |
rag_ | connector_, outcome | Scanned PDF pages sent to OCR |
rag_, rag_ | status | Embedding batches and documents |
rag_, rag_, rag_ | search_ | Searches, their end-to-end duration, and results returned |
rag_ | lane | Search lanes cut by the database statement timeout |
rag_ | result | Chat-answer quotes checked against their cited source |
rag_ | connector_, status | Permission sync passes |
rag_permission_sync_* | connector_ | Permission sync gaps: group failures, dropped principals, ACL over-approximations, unreadable containers, restriction fallbacks, and skipped identity lookups |
rag_ | kind | Users whose group memberships exceeded the per-user cap at query time |
rag_ | Searches limited to organization-wide documents because the caller had no email |
Skill and Sandbox Metrics
| Metric | Labels | Measures |
|---|---|---|
skill_ | activation_ | Skill activations: slash_, chat_, load_, or delegation |
skill_ | activation_ | Tokens that skill activations added to model context |
sandbox_, sandbox_ | status | Code Sandbox commands: ok, script_, timeout, or runtime_ |
sandbox_ | code | Sandbox runtime errors. engine_ means the sandbox engine is down. |
sandbox_ | status | 1 for the runtime's current status: disabled, initializing, ready, error, or stopped |
Background Task Metrics
| Metric | Labels | Measures |
|---|---|---|
task_, task_ | task_ | Tasks queued and completed |
task_ | task_ | Failed attempts. The task may be retried. |
task_ | task_ | Tasks that ran out of retries |
task_ | task_ | Task duration |
task_ | task_ | Tasks running now |
task_ | Stuck tasks returned to the queue | |
schedule_ | agent_, status | Scheduled task runs: success, failed, or cancelled |
Common task_ values are connector_, batch_, permission_, and schedule_.
Platform Metrics
| Metric | Labels | Measures |
|---|---|---|
http_, http_ | method, route, status_ | API request duration, as a histogram and as a summary with quantiles |
database_ | state | Database pool connections: total, idle, and waiting |
database_ | Maximum pool size per process | |
audit_ | source, resource_ | Audit log rows that failed to save |
file_ | provider, scope | Stored files left behind by a failed delete |
chat_ | feedback | Thumbs up, thumbs down, and cleared ratings on chat answers |
process_*, nodejs_* | Standard Node.js process metrics: CPU, memory, heap, event loop lag, and garbage collection |
Example Queries
# Billed LLM spend per agent over the last day
sum by (agent_name) (increase(llm_cost_total{billing_mode="metered"}[1d]))
# 95th percentile LLM latency per model
histogram_quantile(0.95, sum by (le, model) (rate(llm_request_duration_seconds_bucket[5m])))
# MCP tool error rate per server
sum by (mcp_server_name) (rate(mcp_tool_calls_total{status="error"}[5m]))
/ sum by (mcp_server_name) (rate(mcp_tool_calls_total[5m]))
Guardrails and OpenAPPA
Track Guardrails hooks and OpenAPPA engine operations through Archestra's standard Prometheus /metrics endpoint.
Collection Setup
Scrape every Archestra web and worker pod using the standard metrics endpoints and authentication. Hook and engine metrics are included in the same response. No additional exporter, receiver, or data source is required.
If your collector filters metric names, retain openappa_hook_*, appa_*, and agent_, including histogram _bucket, _sum, and _count series. Preserve the metric labels and pod identity labels.
Exercise a Guardrails-enabled request, then query sum by (hook, decision) (openappa_hook_dispatches_total) and sum by (outcome) (appa_policy_decisions_total) in Prometheus. Select that Prometheus source as Metrics on the Guardrails & OpenAPPA dashboard. Metrics appear when their corresponding operations run; an absent event-driven series is not a successful zero.
Embedded Hook Metrics
Archestra exposes two metric families on its standard Prometheus scrape endpoint (:9050/metrics on web pods and :9000/metrics on worker pods; see Scraping).
| Metric | Type | Labels | Measures |
|---|---|---|---|
openappa_ | Counter | hook, decision | Completed embedded OpenAPPA hook dispatches, including failures; one observation after policy-move retries. |
openappa_ | Histogram | hook, decision | Embedded OpenAPPA hook dispatch wall time in seconds (0.001 to 300s buckets), including policy resolution inside dispatch, native initialization, storage, consults, and retries. |
Dispatch Semantics
Each observation records one logical hook dispatch across the embedded host boundary after any policy-move retries settle. One user request or agent turn can trigger multiple hook dispatches (such as session_, tool_, prompt, tool_, and turn_), and a single tool_ dispatch can evaluate policy rules, consult helpers or annotators, and retry when an organization's policy revision moves. These metrics do not count individual engine rule checks or unique user requests.
Metric collection cannot alter enforcement. If metric recording is disabled or fails, Archestra preserves the original hook decision or error unchanged.
Labels
Both families use closed label sets to keep cardinality bounded and never emit organization, session, user, agent, or call IDs, tool names, arguments, policy text, or error messages.
| Label | Values |
|---|---|
hook | session_, prompt, tool_, tool_, turn_, cancel_, child_, child_, child_, yell, or other for an unrecognized hook |
decision | ack, allow_, pass_, deny_, block, replace_, deliver_, child_, context, refuse, mcp_, other for an unrecognized decision, or error when the dispatch rejects |
Related boundary counters on other pages track downstream effects: llm_ counts tool calls blocked at the LLM Proxy, and agent_ counts runtime boundary refusals.
Shared policy preloading for a tool-call batch and principal lookup happen before this boundary and are excluded.
Query dispatch rates, failures, and tail latency with PromQL:
# Embedded hook dispatch rate by hook and decision
sum by (hook, decision) (
rate(openappa_hook_dispatches_total[5m])
)
# Failed dispatches or structured runtime refusals
sum by (hook, decision) (
rate(openappa_hook_dispatches_total{decision=~"error|refuse"}[5m])
)
# 95th-percentile hook dispatch wall time by hook
histogram_quantile(
0.95,
sum by (le, hook) (rate(openappa_hook_dispatch_duration_seconds_bucket[5m]))
)
Engine Metrics
These metrics describe policy checks, external calls, remedies, and runtime failures inside Archestra. They use the same endpoint as hook metrics. Durations are in seconds. Counters and histograms are cumulative for each process; scrapes do not reset them.
| Metric | Type | Labels | Measures |
|---|---|---|---|
appa_ | Counter | outcome (allowed, denied) | Completed tool-proposal policy decisions |
appa_ | Histogram | outcome (allowed, denied, error) | Policy-check duration, including failed evaluations |
appa_ | Counter | role, outcome (answered, no_) | External helper, annotator, or authority calls |
appa_ | Histogram | role, outcome (answered, no_) | External-call duration |
appa_ | Counter | outcome (executed, declined, no_, refused) | Remedy attempts |
appa_ | Histogram | outcome (executed, declined, no_, refused) | Remedy duration |
appa_ | Counter | component, error_ | Runtime failures classified by bounded component and error type |
A host hook acknowledgment is distinct from a tool-proposal policy decision. One hook dispatch can include multiple engine operations. Compare these counters by their documented operation rather than treating them as interchangeable request counts.
Only operational labels are exported. Organization, user, session, tool, policy, and request content are excluded. If engine collection fails, Archestra continues serving its other process metrics.