aws / aws/aws-durable-execution-sdk-python
[Feature]: Exit gracefully with PENDING when a checkpoint response has no CheckpointToken
- Dominant language
- Python
- Stars
- 53
- Forks
- 25
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 37
Description
### What would you like?
When a checkpoint response arrives without a `CheckpointToken`, the SDK should treat it as a signal that the service will accept no further checkpoints from this invocation. The SDK should stop issuing checkpoints and end the invocation cleanly with `Status: PENDING`.
Why PENDING and not FAILED or a thrown error:
- A missing token does not mean the execution is finished. It means this invocation cannot make further progress. The invocation result must not claim the execution finished.
- A thrown error is an invocation failure. Lambda retries it, but the retry has no valid token and can do nothing useful. It also puts an error in customer logs for a condition the SDK understood.
- PENDING is already what the SDK returns for every suspend (wait, scheduled retry, pending callback). This is a suspend: the invocation is done for now and the execution continues later.
### Current behavior
The checkpoint loop already detects a missing token on a non-empty batch (`state.py`, `_process_checkpoint_batch`), but it reads the missing token as "the execution reached a terminal state":
```python
if output.checkpoint_token:
current_checkpoint_token = output.checkpoint_token
elif updates:
execution_completed = True
```
It then calls `_settle_after_execution_completed()`, which stops checkpointing and settles every queued operation with `OrphanedChildException`. The handler thread keeps running. On its next durable call, `_reject_if_execution_completed` raises `OrphanedChildException`. That class extends `BaseException`, and `execution.py` has no handler for it. So the exception propagates out of the Lambda handler as an unhandled error. The invocation fails; the SDK never returns PENDING. If the handler needs no further durable call it returns SUCCEEDED normally.
So Python already has the detection point. What is missing is the classification: a missing token on a non-empty batch is not always "execution completed"; it can also mean "no further checkpoints from this invocation, execution continues later."
### Possible Implementation
1. At the `elif updates:` branch, treat the missing token as a suspend rather than as "execution completed". A completed execution never invokes the handler again, so answering PENDING for it is harmless; a suspended one needs PENDING.
2. Raise `SuspendExecution` (or a new sibling) in the handler thread instead of `OrphanedChildException` when the missing token is observed. `execution.py` already maps `SuspendExecution` to `InvocationStatus.PENDING`.
3. Keep `_settle_after_execution_completed` semantics for queued operations: they cannot be checkpointed, so they are abandoned and replay on the next invocation. AT_MOST_ONCE steps whose START landed in the last accepted checkpoint will not run again, which is the defined semantics of AT_MOST_ONCE.
4. Unit test: a checkpoint response without `checkpoint_token` on a non-empty batch produces `status: PENDING`, no further checkpoint calls, and no raised exception.
5. Testing SDK: the local runner needs a way to omit the token on a chosen checkpoint so customers can test their handlers against this path.
### Is this a breaking change?
No. The SDK's response set is unchanged; a new internal termination reason is added.
### Additional Context
- Prerequisite: #721 (message-case bug in stale-token classification). That fix is the fallback path whenever the missing token is not detected, so it should land first.
- Sibling issues in the JS and Java SDKs are linked in a comment below.
Contributor guide
Research direction
Start in state.py at _process_checkpoint_batch and inspect execution.py for SuspendExecution and the existing PENDING mapping. Add or update the unit test for a non-empty batch whose checkpoint response lacks checkpoint_token, then check the local runner's checkpoint controls. Done means the invocation returns PENDING, makes no further checkpoint calls, and raises no unhandled exception.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- backend, cloud
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 62/100