[DurableExecution]: Invalid checkpoint token misclassified as terminal; execution fails instead of invocation
- Dominant language
- C#
- Stars
- 1.7k
- Forks
- 503
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 18
Description
### Describe the bug
`Amazon.Lambda.DurableExecution` matches the stale checkpoint token rejection with a case-sensitive prefix check against `"Invalid Checkpoint Token"`. The service emits `"Invalid checkpoint token"` (lowercase `c` and `t`). The check never matches, so a stale token is treated as a terminal 4xx and the whole durable execution fails instead of just the invocation.
### Regression Issue
- [ ] Select this option if this issue appears to be a regression.
### Expected Behavior
When the service rejects a checkpoint call with `InvalidParameterValueException` because the checkpoint token is stale (a newer invocation superseded this one), `IsTerminalCheckpointError` returns `false`. The exception escapes to the host, Lambda retries the invocation, the service issues a fresh token, and the execution continues.
### Current Behavior
`IsTerminalCheckpointError` returns `true`. The catch in `DurableFunction.FunctionHandlerAsync` returns `Status = InvocationStatus.Failed`. The execution is failed every time a stale token is seen.
Root cause, in `Libraries/src/Amazon.Lambda.DurableExecution/DurableFunction.cs`:
https://github.com/aws/aws-lambda-dotnet/blob/8ed93ce74341fc51d4725e0668b5a0cce4fd8e3a/Libraries/src/Amazon.Lambda.DurableExecution/DurableFunction.cs#L202-L207
```csharp
if (ex.ErrorCode == "InvalidParameterValueException"
&& ex.Message != null
&& ex.Message.StartsWith("Invalid Checkpoint Token", StringComparison.Ordinal))
{
return false;
}
```
The service constant (`ValidationErrorMessages.INVALID_CHECKPOINT_TOKEN`, internal) is `"Invalid checkpoint token"`.
The existing test uses the SDK's string rather than the service's, so it cannot catch this:
https://github.com/aws/aws-lambda-dotnet/blob/8ed93ce74341fc51d4725e0668b5a0cce4fd8e3a/Libraries/test/Amazon.Lambda.DurableExecution.Tests/DurableFunctionTests.cs#L488
### Reproduction Steps
1. Construct an `AmazonServiceException` with `ErrorCode = "InvalidParameterValueException"`, `StatusCode = 400`, and message `"Invalid checkpoint token: ..."` (the real service message).
2. Call `IsTerminalCheckpointError` with it (or run a workflow whose checkpoint flush throws it).
3. Observe it returns `true` and the invocation output is `Failed`.
### Possible Solution
1. Change the literal to `"Invalid checkpoint token"`, or use `StringComparison.OrdinalIgnoreCase` to protect against future drift.
2. Update `DurableFunctionTests` to use the real service message.
### Additional Information/Context
The same defect exists in the JS, Python, Java, Rust, and Go durable execution SDKs; each has its own issue. Links are in a comment below.
### AWS .NET SDK and/or Package version used
Amazon.Lambda.DurableExecution 2.0.0 (master at 8ed93ce)
### Targeted .NET Platform
Any
### Operating System and version
Any
Contributor guide
Assessment
This issue has not been assessed yet.