You are viewing the docs for Archestra v1.4. View the latest docs
Observability
Export production telemetry to your existing monitoring stack: Prometheus metrics, OpenTelemetry distributed traces and logs, and web UI user sessions.
Archestra tracks model token usage, latency, execution costs, tool call failures, and background worker queues across every agent run.
| Signal | Protocol | Destination | Configuration |
|---|---|---|---|
| Metrics | Prometheus | Scraped from /metrics | Enabled by default |
| Traces and Logs | OTLP HTTP | Pushed to OpenTelemetry collector | ARCHESTRA_ |
| Real User Monitoring | OTLP HTTP | Browser events forwarded to collector | ARCHESTRA_ (Enterprise) |
Scraping Prometheus Metrics
Each Archestra process serves Prometheus metrics on a dedicated port:
- Web pods:
http://<pod>:9050/metrics. Port configurable withARCHESTRA_.METRICS_ PORT - Worker pods:
http://<pod>:9000/metrics. Knowledge sync and background queue metrics emit from worker pods. Both endpoints include embedded Guardrails hook and engine metrics.
To require authentication on metric scrapes, set ARCHESTRA_ to require an Authorization: Bearer <secret> header. See the full list in the Metrics Reference.
Distributed Tracing and Logs
Export distributed spans and structured logs to any OpenTelemetry collector:
ARCHESTRA_OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
Archestra pushes traces to /v1/traces and backend logs to /v1/logs under service name Archestra. Every LLM generation and tool call forms a span containing model names, token counts, costs, and caller identity. See Tracing.
Grafana Dashboards
Archestra publishes eight prebuilt Grafana dashboards. Core operational dashboards require Prometheus only.
| Dashboard | Key Visualizations | Data Sources |
|---|---|---|
LLM Proxy (archestra-genai-observability) | Proxy and model router outcomes, upstream and streaming latency, token rates, and prompt-cache usage | Prometheus (optional Tempo for exemplars) |
| Costs & Limits | Metered and subscription spend, prompt-cache savings, limit rule evaluations, storage errors, and recent exhausted allowances | Prometheus |
| Agent Runtime | Fleet-wide run lifecycle, provisioning latency, steering, completion delivery, boundary refusals, task health, and optional Kubernetes pod resources | Prometheus (optional Kubernetes Prometheus with cAdvisor and kube-state-metrics) |
| Guardrails / OpenAPPA | Embedded OpenAPPA hook dispatches and latency by hook and decision, runtime boundary refusals and engine operations | Prometheus |
MCP Monitoring (archestra-mcp-monitoring) | MCP Gateway tool call rates, error counts, execution duration, payload sizes, and self-hosted server deployment status | Prometheus (optional Tempo for exemplars) |
Knowledge Base Operations (archestra-rag-knowledge-base) | Connector syncs, document ingestion, OCR, embedding batches, search latency, quote verification, and permission sync gaps | Prometheus |
Platform Health (archestra-application-metrics) | HTTP throughput, Node.js memory and event loop lag, database connection pool, background task queue, and PostgreSQL health | Prometheus |
Session Investigation (archestra-agent-sessions) | Per-session timeline, LLM calls, MCP tool invocations, trace waterfalls, and correlated logs across Agent Runtime and external client sessions | Tempo; optional Loki |
Import or update the dashboards with a Grafana service account that can write dashboards and folders and read/query the selected data sources. Basic authentication also works with GRAFANA_ and GRAFANA_:
GRAFANA_URL=https://grafana.example.com GRAFANA_TOKEN=glsa_xxx \
bash <(curl -sL https://raw.githubusercontent.com/archestra-ai/archestra/release/1.4/platform/dev/grafana/install-dashboards.sh)
The installer is idempotent and upserts every dashboard into the Archestra folder (archestra-dashboards) using stable UIDs—including archestra-genai-observability for LLM Proxy, archestra-application-metrics for Platform Health, and archestra-agent-sessions for Session Investigation—so existing dashboard links stay intact. For external databases on Platform Health, pass --postgres-provider otel (or cloudsql, azure; defaults to helm for the Bitnami postgres_ sidecar). To link upstream latency spikes on LLM Proxy and MCP Monitoring to Tempo traces, enable Prometheus exemplars with --enable-feature=exemplar-storage.
To preselect data sources during import, pass their Grafana UIDs:
GRAFANA_URL=https://grafana.example.com GRAFANA_TOKEN=glsa_xxx \
./install-dashboards.sh --metrics-datasource prometheus \
--traces-datasource tempo --logs-datasource loki
--kubernetes-datasource selects a separate Prometheus source for optional Kubernetes rows and defaults to the metrics UID.
Data Sources and Metric Semantics
Model and agent filters on LLM Proxy affect upstream usage; logical request outcomes have their own filters. Costs & Limits has separate cost and limit filters, with Cost breakdown selecting the grouping dimension. Scrape job selects a Prometheus target group; leave All selected unless you need one group of processes.
Each dashboard scopes its optional data sources and aggregates multi-replica metrics according to how Archestra exports them:
- Session timelines vs. runtime fleet health: Session Investigation inspects individual conversations—working for both Agent Runtime runs and external client sessions that propagate session context via
X-Archestra-Session-IdorX-Appa-Session-ID—and adds trace waterfalls and span logs when Tempo and Loki are configured. Agent Runtime tracks fleet, workload, and pod health rather than a per-session event timeline. - Optional Kubernetes row on Agent Runtime: The Kubernetes resource panels require cAdvisor and kube-state-metrics. Scrape kubelet
/metrics/cadvisorand the kube-state-metrics endpoint into that source, preservingcluster,namespace,pod, andcontainerlabels. Verifykube_andpod_ status_ phase container_return runtime pod series before expanding the row. The panels use an independent Kubernetes Prometheus data source variable and Cluster, Namespace, and Runtime pod name regex filters so you can target your runtime pods without editing queries.memory_ working_ set_ bytes - Limit history: These panels require a metrics source that exports limit-check observations. Archestra 1.4 does not export those observations, so the limit-history panels remain empty with a 1.4-only source.
- Replica-wide health gauges: Fleet health reported identically by every replica (such as
agent_,runtime_ health_ tasks agent_) aggregate withruntime_ health_ age_ seconds maxacross replicas, neversum. - Guardrails and OpenAPPA metrics: Hook dispatches and engine operations come from the same Archestra
/metricsendpoint and use the dashboard's Metrics and Scrape job selections. See the metric reference for their distinct counting boundaries.
Real User Monitoring (RUM)
Export frontend user session activity, page load timings, and feature usage to your collector as OTLP log records.
RUM is an enterprise feature. See Pricing Model.
- Set the collector endpoint in your environment:
bash
ARCHESTRA_RUM_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318 - If your collector requires authentication, set
ARCHESTRA_, or bothRUM_ EXPORTER_ OTLP_ AUTH_ BEARER ARCHESTRA_andRUM_ EXPORTER_ OTLP_ AUTH_ USERNAME ARCHESTRA_.RUM_ EXPORTER_ OTLP_ AUTH_ PASSWORD - Restart the backend to initialize the telemetry pipeline.
Browser events route securely through the Archestra backend and arrive at your collector under the service name Archestra Web. Events never contain prompt text, messages, email addresses, or secrets.
Control ingest volume with ARCHESTRA_ (0.0 to 1.0) and ARCHESTRA_.