Observability
Monitor your instance in your existing stack. For individual requests and conversations inside Archestra, use Logs.
| Need | Set up |
|---|---|
| Request rates, latency, costs, guardrails, and infrastructure health | Prometheus metrics. |
| Follow a model request or tool call across services | OpenTelemetry traces and backend logs. |
| Administrative changes sent to a SIEM | Audit log export. |
| Frontend sessions and page timings | Real User Monitoring. |
Grafana Dashboards
Archestra publishes eight prebuilt Grafana dashboards. Core operational dashboards require Prometheus only.
| Dashboard | Key Visualizations | Data Sources |
|---|---|---|
LLM Proxy (archestra-genai-observability) | Proxy and unified model API outcomes, upstream and streaming latency, token rates, and prompt-cache usage | Prometheus (optional Tempo for exemplars) |
| Costs & Limits | Metered and subscription spend, prompt-cache savings, limit rule evaluations, storage errors, and recent exhausted allowances | Prometheus |
| Agent Runtime | Fleet-wide run lifecycle, provisioning latency, steering, completion delivery, boundary refusals, task health, and optional Kubernetes pod resources | Prometheus (optional Kubernetes Prometheus with cAdvisor and kube-state-metrics) |
| Guardrails / OpenAPPA | Embedded OpenAPPA hook dispatches and latency by hook and decision, runtime boundary refusals and engine operations | Prometheus |
MCP Monitoring (archestra-mcp-monitoring) | MCP Gateway tool call rates, error counts, execution duration, payload sizes, and self-hosted server deployment status | Prometheus (optional Tempo for exemplars) |
Knowledge Base Operations (archestra-rag-knowledge-base) | Connector syncs, document ingestion, OCR, embedding batches, search latency, quote verification, and permission sync gaps | Prometheus |
Platform Health (archestra-application-metrics) | HTTP throughput, Node.js memory and event loop lag, database connection pool, background task queue, and PostgreSQL health | Prometheus |
Session Investigation (archestra-agent-sessions) | Per-session timeline, LLM calls, MCP tool invocations, trace waterfalls, and correlated logs across Agent Runtime and external client sessions | Tempo; optional Loki |
Import or update the dashboards with a Grafana service account that can write dashboards and folders and read/query the selected data sources. Basic authentication also works with GRAFANA_ and GRAFANA_:
GRAFANA_URL=https://grafana.example.com GRAFANA_TOKEN=glsa_xxx \
bash <(curl -sL https://raw.githubusercontent.com/archestra-ai/archestra/main/platform/dev/grafana/install-dashboards.sh)
The installer is idempotent and upserts every dashboard into the Archestra folder (archestra-dashboards) using stable UIDs—including archestra-genai-observability for LLM Proxy, archestra-application-metrics for Platform Health, and archestra-agent-sessions for Session Investigation—so existing dashboard links stay intact. For external databases on Platform Health, pass --postgres-provider otel (or cloudsql, azure; defaults to helm for the Bitnami postgres_ sidecar). To link upstream latency spikes on LLM Proxy and MCP Monitoring to Tempo traces, enable Prometheus exemplars with --enable-feature=exemplar-storage.
To preselect data sources during import, pass their Grafana UIDs:
GRAFANA_URL=https://grafana.example.com GRAFANA_TOKEN=glsa_xxx \
./install-dashboards.sh --metrics-datasource prometheus \
--traces-datasource tempo --logs-datasource loki
--kubernetes-datasource selects a separate Prometheus source for optional Kubernetes rows and defaults to the metrics UID.
Data Sources and Metric Semantics
Model and agent filters on LLM Proxy affect upstream usage; logical request outcomes have their own filters. Costs & Limits has separate cost and limit filters, with Cost breakdown selecting the grouping dimension. Scrape job selects a Prometheus target group; leave All selected unless you need one group of processes.
Each dashboard scopes its optional data sources and aggregates multi-replica metrics according to how Archestra exports them:
- Session timelines vs. runtime fleet health: Session Investigation inspects individual conversations—working for both Agent Runtime runs and external client sessions that propagate session context via
X-Archestra-Session-IdorX-Appa-Session-ID—and adds trace waterfalls and span logs when Tempo and Loki are configured. Agent Runtime tracks fleet, workload, and pod health rather than a per-session event timeline. - Optional Kubernetes row on Agent Runtime: The Kubernetes resource panels require cAdvisor and kube-state-metrics. Scrape kubelet
/metrics/cadvisorand the kube-state-metrics endpoint into that source, preservingcluster,namespace,pod, andcontainerlabels. Verifykube_andpod_ status_ phase container_return runtime pod series before expanding the row. The panels use an independent Kubernetes Prometheus data source variable and Cluster, Namespace, and Runtime pod name regex filters so you can target your runtime pods without editing queries.memory_ working_ set_ bytes - Historical limit detail vs. live headroom: On Costs & Limits,
llm_,limit_ last_ exceeded_ timestamp_ seconds llm_, andlimit_ last_ exceeded_ threshold llm_report process-local observations from the last 24 hours at the time a check was exhausted, not live budget headroom.limit_ last_ exceeded_ usage - Replica-wide health gauges: Fleet health reported identically by every replica (such as
agent_,runtime_ health_ tasks agent_) aggregate withruntime_ health_ age_ seconds maxacross replicas, neversum. - Guardrails and OpenAPPA metrics: Hook dispatches and engine operations come from the same Archestra
/metricsendpoint and use the dashboard's Metrics and Scrape job selections. See the metric reference for their distinct counting boundaries.
Real User Monitoring (RUM)
Frontend monitoring requires an Enterprise license.
- Set the collector endpoint in your environment:
bash
ARCHESTRA_RUM_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318 - If your collector requires authentication, set
ARCHESTRA_, or bothRUM_ EXPORTER_ OTLP_ AUTH_ BEARER ARCHESTRA_andRUM_ EXPORTER_ OTLP_ AUTH_ USERNAME ARCHESTRA_.RUM_ EXPORTER_ OTLP_ AUTH_ PASSWORD - Restart the backend to initialize the telemetry pipeline.
Browser events route securely through the Archestra backend and arrive at your collector under the service name Archestra Web. Events never contain prompt text, messages, email addresses, or secrets.