How do I design eventual consistency with the Saga Pattern in a distributed system?

Short answer: avoiding 2PC is the right call — it blocks and doesn’t scale. A Saga turns one atomic transaction into a sequence of local transactions, each with a compensating action. For your flow, an orchestrator-based Saga is the right choice.

The real issue is this: you can no longer roll everything back inside one transaction. Instead, you have to enforce the discipline of “if a step breaks, compensate the steps that came before it.”

1. Let a central coordinator drive the steps over a queue. The orchestrator (Order Service) pushes commands to the queue: reserve payment → decrement stock → create shipment. Each service reports back when it finishes its work, and the coordinator moves to the next step. The flow lives in one place — explicit and readable.
2. On failure, run compensations in reverse. If the shipping step blows up, the orchestrator walks backwards: restock the items, refund the payment. Compensating operations aren’t an “undo,” they are forward-direction balancing operations — a refund is a real transaction, not a magic rollback.
3. Make every step idempotent. Retries are guaranteed on queues; the same “decrement stock” command can arrive twice. Put a deterministic saga_id + step_id on each command and skip it if already processed. Without idempotency, a saga produces double refunds / double stock on the second attempt.
4. Preserve atomicity with the outbox pattern. The pair “do local work + publish the next event” must happen in one transaction. Write the work and the event to an outbox table in the same DB, and let a separate relay move it to the queue. Otherwise the work commits but the event doesn’t publish, and the saga is left stranded.
5. Persist the saga state. Which step you’re on, which ones completed — keep it in a table so that even if the coordinator crashes, it resumes where it left off when it comes back up.

Bottom line: prefer orchestration over choreography; the flow is explicit and the compensation ordering is clear. Accept eventual consistency: stock may look inconsistent for a few seconds, then reconcile. I go deeper on the architecture on sade.dev and on the queue mechanics in the hub post.