Durable Execution
Durable execution is a programming model where a workflow's progress is automatically persisted, so if the process crashes, restarts, or the machine dies, the workflow resumes exactly where it left off — with local variables, call stack, and completed steps intact. You write ordinary-looking code; the engine makes it survive failures that would destroy a normal program.
It targets a problem every backend eventually hits: reliably running long, multi-step processes — an order fulfillment that spans payment, inventory, shipping, and email; a user onboarding with waits and approvals; a data pipeline with retries. Doing this robustly by hand means a sprawl of queues, state tables, cron jobs, idempotency keys, and retry logic — and it's still fragile. Durable execution engines (Temporal is the best-known) collapse that mess into code that reads sequentially but is crash-proof underneath.
TL;DR
- Durable execution persists a workflow's progress automatically so it survives crashes and resumes from the exact point of failure — no lost state, no restart from zero.
- You write workflows as code that looks sequential; the engine records every step so it can replay and continue after any failure.
- It replaces the hand-built sprawl of queues + state tables + cron + retry logic for long-running, multi-step processes.
- Core split: workflows (deterministic orchestration logic, replayed from history) and activities (the actual side-effecting work — API calls, DB writes — that can fail and retry).
- Workflows must be deterministic — the engine replays their history to rebuild state, so no random values, wall-clock time, or direct I/O inside workflow code.
- Ideal for sagas, human-in-the-loop, scheduled and long-waiting processes; overkill for a simple request/response endpoint.
Quick Example
A durable workflow reads like a normal function — but each step is persisted, so a crash mid-way resumes at the next unfinished step, not the beginning. In Temporal-style pseudocode:
That sleep(days=1) is not a thread blocked for a day — the workflow is persisted and rehydrated a day later. The process can be redeployed, crash, and restart in between; the workflow continues as if nothing happened.
Core Concepts
The Problem It Solves
Consider "charge the card, reserve stock, wait a day, then ship." In a normal service this is perilous: if the process crashes after charging but before reserving, what happens? You end up building state tables to track where each order is, queues to hand off steps, cron jobs to resume waits, idempotency keys so retries don't double-charge, and reconciliation to catch what fell through the cracks. It's a lot of undifferentiated plumbing, and it's easy to get subtly wrong.
Durable execution provides that reliability as a platform primitive. You express the process as sequential code; the engine guarantees it runs to completion despite crashes, restarts, and deploys.
How It Works: Event History and Replay
The engine records an event history of everything the workflow does — each step started, each result returned, each timer set. When a workflow needs to resume (after a crash, or to continue after a wait), the engine replays that history to deterministically reconstruct the workflow's exact state — local variables, position in the code — and then continues from the next unfinished step. Completed steps aren't re-run; their recorded results are replayed. This replay mechanism is the magic, and it's why the determinism rule exists.
Workflows vs Activities
The model splits code into two kinds:
Workflows decide what happens and in what order; activities do the fallible, side-effecting work. Activities get automatic retries with backoff; workflows get durability through replay.
The Determinism Constraint
Because a workflow is rebuilt by replaying its history, its code must produce the same decisions given the same history — it must be deterministic. That means inside workflow code: no random(), no reading the wall clock directly, no direct network/DB calls, no non-deterministic iteration. Anything non-deterministic goes in an activity (whose result is recorded and replayed). Engines provide deterministic substitutes (workflow-safe timers, side-effect wrappers). Violating determinism is the classic durable-execution bug.
When to Use It
Durable execution shines for processes that are long-running, multi-step, or must not lose progress:
- Sagas / distributed transactions — multi-service operations needing coordinated steps and compensations (undo a charge if shipping fails). See Event-Driven Architecture.
- Human-in-the-loop — workflows that pause for approval, signature, or input, possibly for days, then resume.
- Scheduled and long-waiting — "wait 30 days then charge renewal," "remind after 3 days if not completed."
- Reliable pipelines — multi-stage data or background jobs where every stage must complete despite failures.
- Orchestrating flaky services — coordinating calls to third-party APIs with built-in retries and timeouts.
It's overkill for a plain synchronous request/response endpoint or a single fire-and-forget task — reach for it when the process is genuinely long, stateful, and failure-sensitive.
Best Practices
Keep Workflow Code Deterministic
All non-determinism — random values, current time, I/O, external calls — belongs in activities, not workflows. Use the engine's workflow-safe timers and side-effect helpers. This is the rule that makes replay work; breaking it causes subtle, hard-to-debug failures on resume.
Make Activities Idempotent
Activities are retried automatically, so an activity may run more than once. Design them to be safe to repeat — use idempotency keys for charges, upserts for writes — so a retry after a partial failure doesn't double-charge or duplicate. See Background Jobs.
Model Compensation for Sagas
In multi-step processes across services, plan the undo path: if step 4 fails, how do you reverse steps 1–3? Durable execution makes implementing the saga's compensation logic straightforward, but you still have to design what "compensate" means for each step.
Version Workflows Carefully
Long-running workflows may be mid-flight when you deploy new code. Changing workflow logic can break replay of in-progress histories. Use the engine's versioning/patching mechanisms to evolve workflow code without corrupting running instances.
Right-Size the Adoption
Durable execution adds an engine and operational surface (Temporal has a server/cluster to run or a managed service to buy). Adopt it where reliability of long, stateful processes justifies it — not for every endpoint. Many apps have a handful of workflows that warrant it and a lot of code that doesn't.
Common Mistakes
Non-Determinism in Workflow Code
Non-Idempotent Activities
An activity that isn't safe to run twice will eventually double-charge or duplicate a record, because retries will happen. Make side effects idempotent.
Using It for Everything
Wrapping a trivial synchronous endpoint in a durable workflow adds latency and operational overhead for no benefit. Reserve durable execution for long-running, multi-step, failure-sensitive processes.
FAQ
What is durable execution, exactly?
It's a model where a workflow's progress is automatically persisted, so the process can crash, restart, or be redeployed and the workflow resumes from the exact step it reached — with its variables and completed steps intact. You write sequential-looking code; the engine records every step and replays history to reconstruct state after any failure. It turns fragile long-running processes into crash-proof ones.
How is Temporal different from a message queue?
A message queue delivers individual messages; you still hand-build the orchestration, state tracking, retries, and timers around them. Durable execution gives you the whole workflow as reliable, sequential code — state, retries, timers, and crash recovery are built in. Queues are a primitive you assemble solutions from; durable execution is the solution for multi-step, stateful processes. Engines like Temporal often use queues internally.
Why must workflow code be deterministic?
Because the engine reconstructs a workflow's state by replaying its recorded history — re-executing the workflow code against past events. If the code made non-deterministic choices (random values, wall-clock time, direct I/O), replay would diverge from the original run and corrupt state. So all non-determinism goes into activities, whose results are recorded and replayed, keeping the workflow itself reproducible.
What's the difference between a workflow and an activity?
A workflow is the orchestration logic — the sequence of steps and decisions — and must be deterministic because it's replayed. An activity is a single unit of actual work (an API call, a DB write, sending email) that can be non-deterministic and can fail; the engine retries activities automatically and records their results. Workflows decide what happens; activities do the fallible, side-effecting work.
When should I NOT use durable execution?
For simple, short request/response endpoints or single fire-and-forget tasks, it's overhead — an extra engine and operational surface for no real benefit. Use it when a process is genuinely long-running, multi-step, stateful, and must not lose progress on failure (sagas, human-in-the-loop, scheduled waits, reliable pipelines). Apply it to the workflows that warrant it, not everywhere.
Related Topics
- Background Jobs — Simpler async work; durable execution handles the complex, stateful cases
- Message Queues — A lower-level primitive durable engines build on
- Event-Driven Architecture — Sagas and coordination patterns
- Microservices — Where distributed workflows and compensation matter most
- Error Handling — Retries, timeouts, and failure design
- Scalability — Running workflows reliably at volume
- Backend Development — The broader context