Production Readiness

Baseline Configuration

• Enable metrics and tracing.
• Use Postgres with tuned pool size.
• Set worker count according to CPU and downstream limits.
• Configure Redis with replication and proper memory policy.
• For larger worker fleets, prefer Redis Cluster plus shard-aware worker polling.

Running Redis Cluster in Production

Use Redis Cluster when one global ready queue becomes a hotspot. Trama spreads queue traffic across virtual shards and uses rendezvous hashing so each worker pod only polls the shards it owns.

• Set redis.topology to CLUSTER on every worker.
• Provide at least three seed nodes in redis.cluster.nodes.
• Use a stable shared redis.queue.keyPrefix and redis.sharding.membershipKey across the fleet.
• Set redis.sharding.podId to a unique pod identity such as ${HOSTNAME}.
• Keep redis.sharding.membershipTtlMillis comfortably above the heartbeat interval.
• Scale by adding worker replicas, not by manually partitioning queue keys.

redis:
  topology: "CLUSTER"
  cluster:
    nodes:
      - "redis://redis-cluster-0.redis:6379"
      - "redis://redis-cluster-1.redis:6379"
      - "redis://redis-cluster-2.redis:6379"
  sharding:
    podId: "${HOSTNAME}"
    virtualShardCount: 64
    membershipKey: "saga:runtime:pods"
    membershipTtlMillis: 10000
    heartbeatIntervalMillis: 3000
    refreshIntervalMillis: 2000
    claimerCount: 4

During shutdown, a worker marks itself as not ready, stops claiming new Redis queue work, unregisters from shard membership, and drains already-claimed executions before closing Redis. That lets another worker take ownership without abandoning in-flight items.

While a worker is processing an execution it keeps renewing its claim (every third of redis.consumer.processingTimeoutMillis, 20 s by default), so slow but healthy executions are never handed to a second worker. If a pod dies without draining, its renewals stop: once the lease expires, the shard's current owner moves that work back to the ready queue on its next claim pass, so recovery takes roughly the lease length.

A worker that pauses past its lease (long GC pause, VM migration, host suspend) is fenced. It stops before its next call, and none of its writes can land, because each one is a compare-and-set on the execution's checkpoint sequence in Postgres. Since 2.1.0.

Scaling Guidelines

• Scale horizontally by adding runtime instances.
• runtime.workerCount (default 32 since 2.2.0) is how many executions a process runs at once, and so how many concurrent calls it makes to downstream services. Workers mostly wait on HTTP: raise it until CPU saturates. On the reference host one process went from 55 workflows/s at 4 workers to 231 at 32.
• Keep database.pool.maxPoolSize in proportion: each completed node writes one checkpoint.
• When running Redis Cluster, tune redis.sharding.virtualShardCount and redis.sharding.claimerCount with replica count.
• Track enqueue/dequeue and p95 saga latency as primary signals.

Resilience Playbook

• Alert on failure ratio and queue pressure.
• Use POST /workflows/{id}/retry for recoverable failures. It only accepts FAILED executions (409 otherwise) and re-runs them from the failed step with the original payload. v1 (steps) definitions only.
• Redis outages. Claim loops back off and resume on their own; /readyz reports consumer_stalled meanwhile. Runs and callbacks are still accepted, because they are written to Postgres and queued once Redis is back.
• Redis data loss (restart without persistence, failover). No execution is lost: the reconciler re-sends anything that stopped advancing, from its Postgres checkpoint, after reconciler.staleAfterMillis (2 min). AOF everysec is recommended to shorten that delay, but it is not required for correctness.
• Postgres outages pause progress; nothing is lost. Watch saga_redis_errors_total and the DB pool metrics.
• When a worker stops heartbeating, its membership entry expires after redis.sharding.membershipTtlMillis and its shards are reassigned on the next refresh; work it had claimed returns to the queue once its lease expires.
• Watch for claims expired before renewal in worker logs: a pod stalled long enough (GC pause, Redis outage) for a claim to expire, so that execution may run twice. Delivery is at-least-once, so keep task endpoints idempotent.

Security Checklist

• Use TLS for ingress and internal service calls.
• Store credentials in secret manager, not plain env in CI logs.
• Restrict network access to Redis/Postgres.
• Harden callback endpoints with authentication/authorization.

Operational Checklist

Upgrading

Read the changelog before upgrading.

Notes for 2.x → 3.0. This release is breaking for JSON body templates:

• JSON bodies use typed replacement only. Whole-value references keep their JSON type, so numbers stop being sent as strings. A missing reference fails the node instead of rendering empty. Triple braces and sections are rejected in JSON bodies. See JSON bodies.
• Check persisted and running workflows before upgrading, and before rolling back. Bodies without templates, or whose references point to nodes that already ran, are unaffected. Workflows that relied on an absent value rendering as empty need explicit initial state. Workflows written for 3.0 (typed collections, literal envelopes) do not render correctly on 2.x.
• The X-Trama-Template-Mode header is ignored. There is no database migration and no configuration change.
• Use 3.0.1 or later. In 3.0.0 and 2.x, a node that ran right after a switch lost its result once the execution resumed after a sleep or callback: 2.x rendered it empty, and 3.0.0 fails with missing key. 3.0.1 reads already stored results correctly.

Notes for 2.3: redis.sharding.virtualShardCount defaults to 64 (was 1024). During a rolling deploy, pods on the old count keep claiming shards 64–1023 until they are replaced, and upgraded pods then move what is left. Set REDIS_SHARDING_VIRTUALSHARDCOUNT=1024 to keep the old count.

Notes for 2.0.x → 2.1 / 2.2:

• Rolling upgrades are supported. Liquibase adds the checkpoint columns on startup (additive changeset 009), and executions started by 2.0.x continue under the new version.
• Deprecated keys are accepted but ignored: redis.pool.* and runtime.bufferSize.
• 2.2.0 raises runtime.workerCount from 4 to 32. That is 8× the concurrent downstream calls per process. Set RUNTIME_WORKERCOUNT=4 to keep the old load.
• Logging (2.2.0). Per-node log lines moved from INFO to DEBUG; each execution now logs one INFO saga finished line.
• Rate limiter (2.1.0). It counts only worker failures, not executions that end FAILED.

Notes for 1.x → 2.x:

• The HTTP API moved from /sagas/* to /workflows/*, including the async callback URL (services that use {{runtime.callback.url}} need no change).
• The shard → pod mapping changed in 2.0.0. During a rolling deploy with mixed versions a shard can briefly have no owner, so prefer a quick rollout or a recreate strategy for that upgrade.
• Liquibase adds the nullable saga_execution.payload column on startup; no manual migration is needed.
• redis.consumer.requeueIntervalMillis is ignored and processingTimeoutMillis now defaults to 20000.