aws / aws/aws-durable-execution-docs
Document step behavior when the Lambda invocation times out mid-step (no StepFailed recorded, retry budget not consumed)
- Dominant language
- Python
- Stars
- 13
- Forks
- 13
- Avg merge
- 3d 2h
- Merged PRs (30d)
- 25
Description
## Summary
The documentation does not explicitly describe what happens when the Lambda
runtime is terminated while a step is still executing — for example, when a
step's work takes longer than the function's invocation timeout (up to the
15-minute Lambda maximum), the sandbox crashes, or the SDK is unable to reach
the checkpoint API.
In this situation:
- The step never completes, so no `StepFailed` event is recorded in the
execution history — only an `InvocationCompleted` event with an error
(e.g. `Sandbox.Timedout`).
- Because the step is not marked as failed, the step's retry strategy /
retry budget is **not** consumed.
- Lambda retries the invocation, replays the execution, and (with default
at-least-once step semantics) re-runs the same incomplete step from the
beginning.
- If the step deterministically takes longer than the invocation timeout,
this repeats indefinitely until the durable execution's `ExecutionTimeout`
is reached or the execution is stopped manually.
This is surprising if you expect a step retry strategy (e.g. "max 3
attempts") to bound how many times a step's code can run. That bound only
applies to failures the SDK observes and checkpoints — not to invocation-level
interruptions.
## What the docs cover today
- The step operation and error-handling pages describe
`StepInterruptedError` for at-most-once steps that Lambda interrupted
before the result was checkpointed.
- The idempotency best-practices page explains at-most-once vs
at-least-once semantics and mentions that a process crash or invocation
timeout can cause a step to run again.
What's missing is an explicit, discoverable treatment of the
**runaway-step failure mode** above and the recommended mitigations.
## Proposed documentation changes
1. **Describe the failure mode explicitly**, ideally in
`patterns/best-practices/step-design.md` (or a dedicated section in the
error-handling docs): when the runtime is terminated mid-step (invocation
timeout, sandbox crash, inability to checkpoint), no `StepFailed` event
is recorded, retry budgets are not consumed, and the step restarts on the
next invocation. Include the concrete consequence: a step whose work
always exceeds the invocation timeout will loop until `ExecutionTimeout`.
2. **Document the mitigations:**
- **Keep step work well under the invocation timeout.** A single step's
execution must fit inside one invocation; split long-running work into
smaller steps, or move it out of the function entirely (e.g.
`invoke`, callbacks).
- **Use at-most-once-per-retry semantics** for steps that might be
interrupted, so the interruption is detected (`StepInterruptedError` /
`StepInterruptedException`) and subject to the retry strategy instead
of silently restarting.
- **Implement an in-step timeout** so the step fails (and checkpoints the
failure) before the sandbox is terminated:
- TypeScript: race the work against a timeout promise and use an
`AbortController` to cancel it.
- Java: `DurableFuture.anyOf(longStep, context.wait(...))` — see the
[WaitExample in the Java SDK](https://github.com/aws/aws-durable-execution-sdk-java/blob/main/examples/src/main/java/software/amazon/lambda/durable/examples/wait/WaitExample.java).
- Python / C#: equivalent cancellation patterns.
3. **Cross-link** the new content from:
- `sdk-reference/operations/step.md` (retry strategy section — clarify
what the retry budget does and does not bound)
- `sdk-reference/error-handling/retries.md`
- `patterns/best-practices/idempotency.md`
- `getting-started/key-concepts.md` (Timeouts section — note that the
function timeout also bounds how long any single step may run)
## Why this matters
Without this guidance, a single misbehaving step can consume the entire
execution timeout in silent retry loops, with nothing in the step-level
history indicating a failure. Users currently have to discover the pattern
themselves (e.g. by reading execution history at startup and detecting
consecutive `InvocationCompleted` errors with no durable progress).
Contributor guide
Research direction
Start with patterns/best-practices/step-design.md, then review the linked sections in sdk-reference/operations/step.md, sdk-reference/error-handling/retries.md, patterns/best-practices/idempotency.md, and getting-started/key-concepts.md. Compare their existing treatment of interruptions, retries, and timeouts; done means the runaway-step behavior, mitigations, and cross-links are clearly documented across the relevant pages.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, csharp, java, python, typescript
- Domain
- cloud, documentation
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100