[DurableExecution]: Return Pending when a checkpoint response has no CheckpointToken
- Dominant language
- C#
- Stars
- 1.7k
- Forks
- 503
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 18
Description
### Describe the feature
When a `CheckpointDurableExecution` response arrives without a `CheckpointToken`, the SDK should treat it as a signal that the service will accept no further checkpoints from this invocation. The SDK should stop issuing checkpoints and end the invocation cleanly with `Status = InvocationStatus.Pending`.
### Use Case
A missing token does not mean the execution is finished. It means this invocation cannot make further progress and the execution continues in a later invocation.
- `Failed` is wrong: it claims the execution finished.
- Letting an exception escape to the host is wrong: Lambda retries the invocation, but the retry has no valid token and can do nothing useful. It also puts an error in customer logs for a condition the SDK understood.
- `Pending` is what the SDK already returns for every suspend (wait, callback, scheduled retry). This is a suspend.
Current behavior (master at 8ed93ce):
`LambdaDurableServiceClient.CheckpointAsync` returns `response.CheckpointToken` unchanged, so a missing token becomes `null`:
https://github.com/aws/aws-lambda-dotnet/blob/8ed93ce74341fc51d4725e0668b5a0cce4fd8e3a/Libraries/src/Amazon.Lambda.DurableExecution/Services/LambdaDurableServiceClient.cs#L88
`CheckpointBatcher` stores it:
https://github.com/aws/aws-lambda-dotnet/blob/8ed93ce74341fc51d4725e0668b5a0cce4fd8e3a/Libraries/src/Amazon.Lambda.DurableExecution/Internal/CheckpointBatcher.cs#L242-L243
The next flush sends `CheckpointToken = checkpointToken ?? ""`. The service rejects the empty token with a 400. `IsTerminalCheckpointError` classifies that 400 as terminal (see the linked bug for the stale-token carve-out; an empty token may also be rejected as a validation error with a different message, which the carve-out would not cover either way). The invocation returns `Failed`. So a missing token fails the execution today.
### Proposed Solution
1. In `CheckpointBatcher` (or the flush delegate), when the returned token is `null`, stop flushing and signal the `TerminationManager` with a new suspend reason. `DurableExecutionHandler.RunAsync` already maps a termination without an exception to `InvocationStatus.Pending`, so no change is needed there.
2. Do not issue further checkpoint calls; there is no token to send. Queued updates are abandoned and replay on the next invocation. AT_MOST_ONCE steps whose START landed in the last accepted checkpoint will not run again, which is the defined semantics of AT_MOST_ONCE.
3. Unit test: a checkpoint response with a null token produces `Pending`, no further checkpoint calls, and no exception.
4. `Amazon.Lambda.DurableExecution.Testing`: allow the in-memory service to omit the token on a chosen checkpoint so customers can test their workflows against this path.
### Other Information
Prerequisite: the stale-token classification bug (linked in a comment below). That fix is the fallback path whenever the missing token is not detected, so it should land first.
The same feature is being tracked in the JS, Python, Java, Rust, and Go durable execution SDKs; links are in a comment below.
### Acknowledgements
- [ ] I may be able to implement this feature request
- [ ] This feature might incur a breaking change
### AWS .NET SDK and/or Package version used
Amazon.Lambda.DurableExecution 2.0.0 (master at 8ed93ce)
### Targeted .NET Platform
Any
### Operating System and version
Any
Contributor guide
Research direction
Start in Libraries/src/Amazon.Lambda.DurableExecution/Internal/CheckpointBatcher.cs and follow the token handling into LambdaDurableServiceClient.CheckpointAsync and DurableExecutionHandler.RunAsync. Add coverage for a null checkpoint token, verifying Pending, no further checkpoint calls, and no exception. Then update Amazon.Lambda.DurableExecution.Testing so an in-memory checkpoint can omit its token.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, csharp
- Domain
- backend, cloud, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100