aws / aws/aws-durable-execution-docs

Document step behavior when the Lambda invocation times out mid-step (no StepFailed recorded, retry budget not consumed)

Open
#281 0 comments 0 reactions 0 assignees View on GitHub
documentation
Dominant language
Python
Stars
13
Forks
13
Avg merge
3d 2h
Merged PRs (30d)
25

Description

## Summary

The documentation does not explicitly describe what happens when the Lambda
runtime is terminated while a step is still executing — for example, when a
step's work takes longer than the function's invocation timeout (up to the
15-minute Lambda maximum), the sandbox crashes, or the SDK is unable to reach
the checkpoint API.

In this situation:

- The step never completes, so no `StepFailed` event is recorded in the
execution history — only an `InvocationCompleted` event with an error
(e.g. `Sandbox.Timedout`).
- Because the step is not marked as failed, the step's retry strategy /
retry budget is **not** consumed.
- Lambda retries the invocation, replays the execution, and (with default
at-least-once step semantics) re-runs the same incomplete step from the
beginning.
- If the step deterministically takes longer than the invocation timeout,
this repeats indefinitely until the durable execution's `ExecutionTimeout`
is reached or the execution is stopped manually.

This is surprising if you expect a step retry strategy (e.g. "max 3
attempts") to bound how many times a step's code can run. That bound only
applies to failures the SDK observes and checkpoints — not to invocation-level
interruptions.

## What the docs cover today

- The step operation and error-handling pages describe
`StepInterruptedError` for at-most-once steps that Lambda interrupted
before the result was checkpointed.
- The idempotency best-practices page explains at-most-once vs
at-least-once semantics and mentions that a process crash or invocation
timeout can cause a step to run again.

What's missing is an explicit, discoverable treatment of the
**runaway-step failure mode** above and the recommended mitigations.

## Proposed documentation changes

1. **Describe the failure mode explicitly**, ideally in
`patterns/best-practices/step-design.md` (or a dedicated section in the
error-handling docs): when the runtime is terminated mid-step (invocation
timeout, sandbox crash, inability to checkpoint), no `StepFailed` event
is recorded, retry budgets are not consumed, and the step restarts on the
next invocation. Include the concrete consequence: a step whose work
always exceeds the invocation timeout will loop until `ExecutionTimeout`.

2. **Document the mitigations:**
- **Keep step work well under the invocation timeout.** A single step's
execution must fit inside one invocation; split long-running work into
smaller steps, or move it out of the function entirely (e.g.
`invoke`, callbacks).
- **Use at-most-once-per-retry semantics** for steps that might be
interrupted, so the interruption is detected (`StepInterruptedError` /
`StepInterruptedException`) and subject to the retry strategy instead
of silently restarting.
- **Implement an in-step timeout** so the step fails (and checkpoints the
failure) before the sandbox is terminated:
- TypeScript: race the work against a timeout promise and use an
`AbortController` to cancel it.
- Java: `DurableFuture.anyOf(longStep, context.wait(...))` — see the
[WaitExample in the Java SDK](https://github.com/aws/aws-durable-execution-sdk-java/blob/main/examples/src/main/java/software/amazon/lambda/durable/examples/wait/WaitExample.java).
- Python / C#: equivalent cancellation patterns.

3. **Cross-link** the new content from:
- `sdk-reference/operations/step.md` (retry strategy section — clarify
what the retry budget does and does not bound)
- `sdk-reference/error-handling/retries.md`
- `patterns/best-practices/idempotency.md`
- `getting-started/key-concepts.md` (Timeouts section — note that the
function timeout also bounds how long any single step may run)

## Why this matters

Without this guidance, a single misbehaving step can consume the entire
execution timeout in silent retry loops, with nothing in the step-level
history indicating a failure. Users currently have to discover the pattern
themselves (e.g. by reading execution history at startup and detecting
consecutive `InvocationCompleted` errors with no durable progress).

Contributor guide

Open the contributing guide

Research direction

Start with patterns/best-practices/step-design.md, then review the linked sections in sdk-reference/operations/step.md, sdk-reference/error-handling/retries.md, patterns/best-practices/idempotency.md, and getting-started/key-concepts.md. Compare their existing treatment of interruptions, retries, and timeouts; done means the runaway-step behavior, mitigations, and cross-links are clearly documented across the relevant pages.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, csharp, java, python, typescript
Domain
cloud, documentation
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.