aws / aws/aws-durable-execution-sdk-python
[Bug]: Emulator retries invalid invocation outputs that the service fails immediately
- Ngôn ngữ chính
- Python
- Star
- 53
- Fork
- 25
- Merge trung bình
- 1 ngày 19 giờ
- Pull request đã merge (30 ngày)
- 40
Mô tả
### Expected Behavior
The emulator should mirror the service's handling of invalid invocation outputs. Verified service behavior:
Most invalid outputs **fail the execution immediately, with no retry**. This applies to:
- `Status=FAILED` with a `Result` present
- `Status=SUCCEEDED` with an `Error` present
- a missing or unrecognized `Status`
- an unparseable/malformed invocation output payload
In each case the service fails the execution with an error object of the form:
```json
{
"ErrorType": "InvalidParameterValueException",
"ErrorMessage": ""
}
```
There is exactly **one retried case**: `Status=PENDING` when the execution has no pending operations. The service treats this as a transient runtime-level error (it can arise from SDK race conditions that resolve on replay) and retries the invocation, up to 3 consecutive attempts, before failing the execution with the same `InvalidParameterValueException`-shaped error.
### Actual Behavior
In `aws_durable_execution_sdk_python_testing.executor.Executor._finish_invocation()`, **all** validation failures from `_validate_invocation_response_and_store()` are routed through the same retry path:
```python
except (InvalidParameterValueException, IllegalStateException) as e:
...
self._set_invocation_gate(execution_arn, InvocationState.PRE_INVOKE)
self._retry_invocation(execution, error_obj)
return
```
`_retry_invocation` re-invokes the handler up to `MAX_CONSECUTIVE_FAILED_ATTEMPTS = 5` times with a flat `RETRY_BACKOFF_SECONDS = 5` backoff before failing.
Consequences:
- A deterministically invalid output (e.g. `SUCCEEDED` with an `Error`) is re-invoked 5 times before failing, where the service fails on the first response. For a deterministic handler this adds ~25 seconds and 4 pointless invocations, and every re-invocation replays the handler.
- The one case the service *does* retry (`PENDING` with no pending operations) gets 5 attempts instead of 3.
- The terminal error surfaced to the customer is `ErrorObject.from_exception(e)` rather than the service's `{ErrorType: "InvalidParameterValueException", ErrorMessage: ...}` shape.
### Proposed Fix
In `_finish_invocation`, distinguish the two classes:
- Fail-fast validation errors (`FAILED`+`Result`, `SUCCEEDED`+`Error`, missing/unknown status, malformed output): fail the execution immediately with an `InvalidParameterValueException`-typed error object carrying the validation message.
- `PENDING` with no pending operations: keep the retry path, with a consecutive-attempt cap of 3.
Regression tests:
- each fail-fast case produces a FAILED execution after a **single** invocation, with the `InvalidParameterValueException` error shape;
- `PENDING` with no pending operations retries and fails after 3 consecutive attempts;
- a transient case (invalid output once, then a valid `PENDING`/`SUCCEEDED`) recovers without failing the execution.
### Related
- Found while investigating https://github.com/aws/aws-durable-execution-sdk-python/issues/656 (FAILED without ErrorObject completed as SUCCEEDED by local emulator). This issue covers the adjacent divergence in the same code path: how invalid outputs are retried vs. failed.
### SDK Version
`aws-durable-execution-sdk-python-testing` current `main` as of 2026-08-21.
### Python Version
3.13
### Is this a regression?
No known working version.
Hướng dẫn đóng góp
Hướng nghiên cứu
Bắt đầu trong aws_durable_execution_sdk_python_testing.executor.Executor._finish_invocation() và lần theo _validate_invocation_response_and_store() cùng _retry_invocation(). Tái hiện các trường hợp đầu ra không hợp lệ được liệt kê và trường hợp PENDING, sau đó thêm kiểm thử hồi quy thể hiện hành vi fail-fast, dạng lỗi InvalidParameterValueException, giới hạn ba lần thử đối với PENDING và khả năng phục hồi sau một đầu ra không hợp lệ tạm thời.
Do mô hình lập chỉ mục viết ra từ nội dung của issue.
Đánh giá
- Công nghệ
- python
- Lĩnh vực
- backend
- Loại issue
- Lỗi
- Độ khó
- 4/5
- Thời gian dự kiến
- 3-5 ngày
- Mức độ hoạt động
- Sôi nổi
- Độ rõ ràng
- Đặc tả rõ ràng
- Mức phù hợp với người mới
- 72/100