Architecture

Why orchestration exists

In microservices systems, business transactions span multiple services with different failure modes. Trama exists to keep retries, compensation, state tracking, and observability in one orchestration layer instead of scattering that logic across every service boundary.

Core Components

• Application: REST endpoints, callback receiver, and runtime bootstrap.
• RuntimeBootstrap: wires queue, consumer, processor, store, and observability.
• SagaExecutionProcessor: worker orchestration, ack/retry control, and rate-limit integration.
• WorkflowExecutor: node dispatch, retry policy decisions, compensation transitions. Delegates to TaskNodeHandler (sync and async HTTP) and SwitchNodeHandler (JSON Logic branching). Split nodes are handled directly: each branch becomes an independent sub-execution enqueued through the same pipeline, and a join node barriers on all of them reaching a terminal state before the parent continues.
• CallbackReceiver: validates HMAC-signed callback tokens and re-enqueues async executions to resume.
• SagaExecutionStore: persistence. Postgres is the source of truth for each execution's state and resume point, step history, parked state and join barriers.
• SagaExecutionRedisConsumer: claims due work from the Redis queue shards this pod owns (rendezvous hashing over live pods), renews claims while they run.
• ExecutionReconciler: re-sends executions that stopped advancing, from their Postgres checkpoint.

State and durability

Since 2.1.0, every node boundary writes the execution's checkpoint to Postgres in one statement: its state, a sequence number, and the step result. Every write is a compare-and-set on that sequence. A worker holding an outdated copy of the execution therefore cannot overwrite newer progress or a final status. Such a copy can come from a redelivered queue item, or from a worker that paused past its claim lease.

• Redis holds the queue, claims and callback nonces, not execution state. If Redis loses data, executions resume from their checkpoint through the reconciler (after reconciler.staleAfterMillis).
• Delivery is at-least-once, outcomes are exactly-once. An HTTP call in flight when its worker crashes or stalls is made again, but no execution ever ends with two outcomes or a wrong one. Make downstream endpoints idempotent (e.g. Idempotency-Key: {{sagaId}}-<nodeId>).
• The guarantees are tested. They are checked with fault injection (Redis restarts, outages and data loss; worker kills and pauses; Postgres cuts) in the validation report.

Execution Lifecycle

1. Definition is validated and stored (or provided inline).
2. The execution is admitted: its row is written to Postgres, then it is enqueued in Redis.
3. The consumer claims due work from the shards this pod owns and hands it to workers (at most workerCount + prefetch at a time).
4. Executor dispatches each node — sync task nodes make HTTP calls immediately; async task nodes fire a request, pause the execution, and wait for a callback.
5. For async nodes: the external service POSTs to the callback endpoint with a signed token. The callback receiver validates the token and re-enqueues the execution to resume from the next node.
6. Switch nodes evaluate JSON Logic expressions and route to the matching branch without making any HTTP call.
7. Split nodes spawn one independent sub-execution per branch (each with its own id and retry/compensation lifecycle) and park the parent until its join node observes every branch reaching a terminal state — nesting works the same way at any depth.
8. On failure, the executor applies the retry or backoff policy, then drives compensation in reverse order if retries are exhausted.
9. Each completed node is checkpointed in Postgres before the next one runs; the final status is written the same way.
10. Metrics and traces are emitted throughout all stages.

Mermaid Diagram

flowchart TB

classDef api fill:#1e293b,stroke:#60a5fa,stroke-width:2px,color:#e5e7eb;
classDef runtime fill:#1e293b,stroke:#4ade80,stroke-width:2px,color:#e5e7eb;
classDef infra fill:#1e293b,stroke:#f59e0b,stroke-width:2px,color:#e5e7eb;
classDef obs fill:#1e293b,stroke:#c084fc,stroke-width:2px,color:#e5e7eb;
classDef external fill:#1e293b,stroke:#fb7185,stroke-width:2px,color:#e5e7eb;

C[Client]:::api

subgraph Trama
direction TB

A[Ktor API]:::api

subgraph Ingress
direction LR
CB[Callback Receiver]:::api
Q[Redis Queue]:::infra
end

P[Saga Processor]:::runtime
E[Saga Executor]:::runtime
ST[Postgres: state + checkpoints]:::infra
R[Reconciler]:::runtime

M[Prometheus Metrics]:::obs
T[OpenTelemetry Traces]:::obs

end

S[External Services]:::external

C --> A
A -->|admit| ST
A --> Q
A --> CB
CB --> Q
Q --> P
P --> E
E --> S
E -->|checkpoint| ST
ST -.-> R
R -->|re-send stalled| Q

S -->|async callback| A

P -.-> M
E -.-> T