aws / aws/aws-lambda-dotnet

[DurableExecution]: Invalid checkpoint token misclassified as terminal; execution fails instead of invocation

Open
#2,571 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
C#
Stars
1.7k
Forks
503
Avg merge
1d 5h
Merged PRs (30d)
18

Description

### Describe the bug

`Amazon.Lambda.DurableExecution` matches the stale checkpoint token rejection with a case-sensitive prefix check against `"Invalid Checkpoint Token"`. The service emits `"Invalid checkpoint token"` (lowercase `c` and `t`). The check never matches, so a stale token is treated as a terminal 4xx and the whole durable execution fails instead of just the invocation.

### Regression Issue

- [ ] Select this option if this issue appears to be a regression.

### Expected Behavior

When the service rejects a checkpoint call with `InvalidParameterValueException` because the checkpoint token is stale (a newer invocation superseded this one), `IsTerminalCheckpointError` returns `false`. The exception escapes to the host, Lambda retries the invocation, the service issues a fresh token, and the execution continues.

### Current Behavior

`IsTerminalCheckpointError` returns `true`. The catch in `DurableFunction.FunctionHandlerAsync` returns `Status = InvocationStatus.Failed`. The execution is failed every time a stale token is seen.

Root cause, in `Libraries/src/Amazon.Lambda.DurableExecution/DurableFunction.cs`:
https://github.com/aws/aws-lambda-dotnet/blob/8ed93ce74341fc51d4725e0668b5a0cce4fd8e3a/Libraries/src/Amazon.Lambda.DurableExecution/DurableFunction.cs#L202-L207

```csharp
if (ex.ErrorCode == "InvalidParameterValueException"
&& ex.Message != null
&& ex.Message.StartsWith("Invalid Checkpoint Token", StringComparison.Ordinal))
{
return false;
}
```

The service constant (`ValidationErrorMessages.INVALID_CHECKPOINT_TOKEN`, internal) is `"Invalid checkpoint token"`.

The existing test uses the SDK's string rather than the service's, so it cannot catch this:
https://github.com/aws/aws-lambda-dotnet/blob/8ed93ce74341fc51d4725e0668b5a0cce4fd8e3a/Libraries/test/Amazon.Lambda.DurableExecution.Tests/DurableFunctionTests.cs#L488

### Reproduction Steps

1. Construct an `AmazonServiceException` with `ErrorCode = "InvalidParameterValueException"`, `StatusCode = 400`, and message `"Invalid checkpoint token: ..."` (the real service message).
2. Call `IsTerminalCheckpointError` with it (or run a workflow whose checkpoint flush throws it).
3. Observe it returns `true` and the invocation output is `Failed`.

### Possible Solution

1. Change the literal to `"Invalid checkpoint token"`, or use `StringComparison.OrdinalIgnoreCase` to protect against future drift.
2. Update `DurableFunctionTests` to use the real service message.

### Additional Information/Context

The same defect exists in the JS, Python, Java, Rust, and Go durable execution SDKs; each has its own issue. Links are in a comment below.

### AWS .NET SDK and/or Package version used

Amazon.Lambda.DurableExecution 2.0.0 (master at 8ed93ce)

### Targeted .NET Platform

Any

### Operating System and version

Any

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.