Observability

Track Trama runtime health with Prometheus metrics, Grafana dashboards, and OpenTelemetry traces across queue claims, failures, latency, and Redis shard ownership. Use this page when you need to answer “is the orchestrator healthy?” and “where are workflows getting stuck?”

Best for SREs Metrics and tracing Production operations

ProductionSee where these signals fit into rollout and scaling decisions. ConfigurationEnable the telemetry and runtime settings behind the metrics. APIMap runtime behavior back to workflow requests and retries.

What you can answer with this data

Use Trama observability to answer whether workflows are flowing through the queue, whether failures are climbing, whether Redis shard ownership is healthy, and whether end-to-end latency is staying within expectations.

Metrics Catalog

Metric	Type	Description
`saga_enqueue_total`	Counter	Queue ingress
`saga_dequeue_total`	Counter	Queue claims
`saga_processed_total`	Counter	Processed outcomes
`saga_failed_total`	Counter	Failures by reason
`saga_retried_total`	Counter	Retry scheduling rate
`saga_rate_limited_total`	Counter	Rate-limited executions
`saga_redis_claim_scans_total`	Counter	Redis shard claim scans. Since 2.1.0, claimers only visit shards with due work, so this tracks real work rather than idle polling.
`saga_redis_errors_total`	Counter	Redis calls that failed in a background loop and were retried with backoff, by `operation` (since 2.1.0)
`saga_fenced_total`	Counter	Work dropped because the worker no longer owned it, by `reason`: `lease_lost` (paused past its claim lease) or `stale_checkpoint` (another copy got ahead). Since 2.1.0.
`saga_reconciled_total`	Counter	Executions re-sent from their Postgres checkpoint after their queue item was lost, by `status` (since 2.1.0)
`saga_inmemory_queue_size`	Gauge	Executions claimed by this process and waiting for a free worker
`saga_redis_active_pods`	Gauge	Healthy worker pods seen in Redis membership.
`saga_redis_owned_shards`	Gauge	Virtual shards currently assigned to this worker.
`saga_redis_membership_refresh_age_ms`	Gauge	Age of the last successful membership refresh.
`saga_duration_seconds`	Histogram	End-to-end duration
`saga_step_duration_success_seconds`	Histogram	Duration of successful task HTTP calls, by `step_name`
`saga_node_duration_seconds`	Histogram	Per-node dispatch duration: the HTTP call for tasks (until accepted for async), condition evaluation for switches, parking for sleeps, fan-out for splits (since 2.0.1)
`saga_callback_received_total`	Counter	Async callbacks delivered to the orchestrator
`saga_callback_rejected_total`	Counter	Async callbacks rejected (expired, bad signature, replay, attempt mismatch)
`saga_callback_timeout_total`	Counter	Async tasks that timed out waiting for a callback
`saga_callback_wait_entered_total`	Counter	Async tasks that started waiting for a callback
`saga_callback_timeout_scan_requeued_total`	Counter	Timed-out waits re-enqueued by the callback timeout scanner
`saga_switch_evaluated_total`	Counter	Switch node evaluations

The Redis cluster rollout adds worker-coordination metrics so you can observe shard ownership, claim pressure, and membership freshness without inspecting Redis directly.

Definition-Level Labels

saga_name, saga_version on counters and histograms.
phase on queue and processing counters.
outcome on processed, reason on failed and callback-rejected counters, result on callback-received.
final_status on saga_duration_seconds.
node_kind (task | switch | sleep | split) and mode (sync | async for tasks, none otherwise) on saga_node_duration_seconds.
result (case | default) on saga_switch_evaluated_total.
Redis membership gauges and the recovery counters (saga_redis_errors_total, saga_fenced_total, saga_reconciled_total) are process-level and do not add definition labels.

Grafana Dashboard

Import grafana/trama-saga-dashboard.json.

Suggested Alerts

Failure ratio above threshold per definition.
p95 saga duration SLO breach.
Sustained enqueue/dequeue imbalance.
Unexpected rate-limit growth.
saga_redis_membership_refresh_age_ms above membership TTL.
saga_redis_active_pods dropping unexpectedly during rollout.
saga_redis_owned_shards equal to 0 on a worker pod that should be active.
Worker log claims expired before renewal: a pod stalled past its claim lease, so an execution may be re-delivered.
saga_redis_errors_total growing: Redis unreachable or failing; /readyz reports consumer_stalled if it lasts.
saga_reconciled_total above zero: queue items were lost (Redis data loss) and executions were recovered with a delay.
saga_fenced_total{reason="lease_lost"} above zero: workers are pausing past their claim lease (GC, CPU starvation).

Logs

Logs are structured JSON (logstash encoder) with the execution id and trace context in the MDC. Since 2.2.0, INFO carries one line per execution, saga finished, with status and durationMs, plus lifecycle events: retries, compensation, sleeps, callback waits and timeouts. The per-node lines (executing node, node completed) are at DEBUG; a failed node still logs node completed at INFO.

Health probes

/readyz: 503 while shutting down, when Redis is unreachable, or when the consumer has made no progress for 3× the claim lease (consumer_stalled).
/healthz: 503 only after runtime.livenessStallMillis (2 min) without consumer progress, so an orchestrator restarts a stuck process.

Tracing

OpenTelemetry spans cover request handling and saga processing when telemetry is enabled.