Observability Metrics

Archestra serves Prometheus metrics in OpenMetrics format. Scrape them to chart and alert on LLM cost and latency, MCP tool calls, Agent Runtime health, and background jobs.

Scraping

Each process serves its own registry, so scrape every pod:

The Helm chart runs the worker as its own Deployment by default. Sum counters across web and worker pods: an agent run from a schedule reports its LLM metrics from a worker pod.

ARCHESTRA_METRICS_PORT changes the web port. Worker pods serve metrics on their API port, 9000.

Authentication

The endpoint is open by default. Set ARCHESTRA_METRICS_SECRET to require a bearer token on web and worker pods. Requests without it get 401.

scrape_configs:
  - job_name: archestra
    authorization:
      type: Bearer
      credentials: <ARCHESTRA_METRICS_SECRET>
    kubernetes_sd_configs:
      - role: pod

Helm Pod Annotations

For a Prometheus that discovers pods by annotation, add these to the chart values:

archestra:
  podAnnotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "9050"
    prometheus.io/path: /metrics

The chart rewrites prometheus.io/port to 9000 on worker pods.

To check a pod, run curl -s http://<pod>:9050/metrics | grep "# HELP llm_". Add -H "Authorization: Bearer <secret>" when a secret is set.

Labels

Most LLM metrics share these labels:

MCP tool call metrics carry agent_id, agent_name, agent_type, mcp_server_name, tool_name, and status (success or error).

Agent Labels

Go to an agent's Advanced tab and add a key and value under Labels. Each agent label becomes an extra label on the LLM, MCP, and agent_runs_total metrics. Prometheus allows only letters, digits, and underscores in label names, so other characters become _: cost-center turns into cost_center.

Dimensions Without Labels

Users, skills, apps, and client-supplied IDs are not metric labels. Use these sources instead:

• Users: per-user statistics, or the archestra.user.* span attributes.
• Skills and apps: per-skill and per-app statistics. App LLM spend shows in the LLM metrics as source="app:llm_complete".
• Client-supplied agent IDs from the X-Archestra-Agent-Id header: the archestra.external_agent_id span attribute. Only agent_runs_total carries it as a label.
• Credentials: the archestra.virtual_key.id span attribute.

LLM Metrics

billing_mode is metered for per-token billing and subscription for flat-rate subscription credentials. Billed spend is sum(llm_cost_total{billing_mode="metered"}). auth_method is provider_key, virtual_key, passthrough_virtual_key, jwks, oauth_client_credentials, oauth_user, internal, or unknown.

llm_time_to_first_byte_seconds minus llm_time_to_first_token_seconds is the time Archestra spends before the upstream call: authentication, guardrails, and tool policies.

Every replica reports the same llm_active_users value, so aggregate it with max(), not sum(). ARCHESTRA_METRICS_ACTIVE_USERS_REFRESH_INTERVAL_MS sets the refresh interval; 0 turns the metric off.

LLM and MCP metrics carry trace exemplars, so a Grafana panel can link a data point to its trace. See Exemplars.

MCP Metrics

mcp_server_deployment_status is 1 for the server's current state: not_created, pending, running, failed, succeeded, hibernated, or waking. count(mcp_server_deployment_status{state="running"} == 1) counts running servers.

Agent Runtime Health

agent_runtime_health_tasks conditions are working, submitted, input_required, auth_required, failed_recent (the last 15 minutes), and completion_pending. Conditions overlap: a task waiting for authentication also counts as working. agent_runtime_health_age_seconds conditions are heartbeat, submitted, and completion_pending.

Every replica reports the same health gauge values, so aggregate them with max, not sum. Alert on failed scrapes too, because missing samples do not mean healthy runs. A fresh heartbeat shows the orchestration is alive, not that the agent is making progress.

This alert finds runs whose heartbeat is more than two minutes old:

max by (agent_id, backend) (
  agent_runtime_health_age_seconds{condition="heartbeat"}
) > 120

Add a pending period (for: 5m) to skip transient spikes.

Knowledge Base Metrics

Skill and Sandbox Metrics

Background Task Metrics

Common task_type values are connector_sync, batch_embedding, permission_sync, and schedule_trigger_run_execute.

Platform Metrics

Example Queries

# Billed LLM spend per agent over the last day
sum by (agent_name) (increase(llm_cost_total{billing_mode="metered"}[1d]))

# 95th percentile LLM latency per model
histogram_quantile(0.95, sum by (le, model) (rate(llm_request_duration_seconds_bucket[5m])))

# MCP tool error rate per server
sum by (mcp_server_name) (rate(mcp_tool_calls_total{status="error"}[5m]))
  / sum by (mcp_server_name) (rate(mcp_tool_calls_total[5m]))