aws / aws/aws-durable-execution-sdk-python

[Bug]: Emulator retries invalid invocation outputs that the service fails immediately

オープン
#670 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
bug parity pkg:testing
主要言語
Python
スター
53
フォーク
25
平均マージ
1日 20時間
マージ済み PR(30日)
39

説明

### Expected Behavior

The emulator should mirror the service's handling of invalid invocation outputs. Verified service behavior:

Most invalid outputs **fail the execution immediately, with no retry**. This applies to:

- `Status=FAILED` with a `Result` present
- `Status=SUCCEEDED` with an `Error` present
- a missing or unrecognized `Status`
- an unparseable/malformed invocation output payload

In each case the service fails the execution with an error object of the form:

```json
{
"ErrorType": "InvalidParameterValueException",
"ErrorMessage": ""
}
```

There is exactly **one retried case**: `Status=PENDING` when the execution has no pending operations. The service treats this as a transient runtime-level error (it can arise from SDK race conditions that resolve on replay) and retries the invocation, up to 3 consecutive attempts, before failing the execution with the same `InvalidParameterValueException`-shaped error.

### Actual Behavior

In `aws_durable_execution_sdk_python_testing.executor.Executor._finish_invocation()`, **all** validation failures from `_validate_invocation_response_and_store()` are routed through the same retry path:

```python
except (InvalidParameterValueException, IllegalStateException) as e:
...
self._set_invocation_gate(execution_arn, InvocationState.PRE_INVOKE)
self._retry_invocation(execution, error_obj)
return
```

`_retry_invocation` re-invokes the handler up to `MAX_CONSECUTIVE_FAILED_ATTEMPTS = 5` times with a flat `RETRY_BACKOFF_SECONDS = 5` backoff before failing.

Consequences:

- A deterministically invalid output (e.g. `SUCCEEDED` with an `Error`) is re-invoked 5 times before failing, where the service fails on the first response. For a deterministic handler this adds ~25 seconds and 4 pointless invocations, and every re-invocation replays the handler.
- The one case the service *does* retry (`PENDING` with no pending operations) gets 5 attempts instead of 3.
- The terminal error surfaced to the customer is `ErrorObject.from_exception(e)` rather than the service's `{ErrorType: "InvalidParameterValueException", ErrorMessage: ...}` shape.

### Proposed Fix

In `_finish_invocation`, distinguish the two classes:

- Fail-fast validation errors (`FAILED`+`Result`, `SUCCEEDED`+`Error`, missing/unknown status, malformed output): fail the execution immediately with an `InvalidParameterValueException`-typed error object carrying the validation message.
- `PENDING` with no pending operations: keep the retry path, with a consecutive-attempt cap of 3.

Regression tests:

- each fail-fast case produces a FAILED execution after a **single** invocation, with the `InvalidParameterValueException` error shape;
- `PENDING` with no pending operations retries and fails after 3 consecutive attempts;
- a transient case (invalid output once, then a valid `PENDING`/`SUCCEEDED`) recovers without failing the execution.

### Related

- Found while investigating https://github.com/aws/aws-durable-execution-sdk-python/issues/656 (FAILED without ErrorObject completed as SUCCEEDED by local emulator). This issue covers the adjacent divergence in the same code path: how invalid outputs are retried vs. failed.

### SDK Version

`aws-durable-execution-sdk-python-testing` current `main` as of 2026-08-21.

### Python Version

3.13

### Is this a regression?

No known working version.

コントリビューションガイド

コントリビューションガイドを開く

調査の方向性

aws_durable_execution_sdk_python_testing.executor.Executor._finish_invocation() から開始し、_validate_invocation_response_and_store() と _retry_invocation() を追跡します。列挙されている無効な出力のケースと PENDING ケースを再現し、その後、fail-fast 動作、InvalidParameterValueException のエラー形状、PENDING の 3 回試行上限、および一時的な無効な出力からの回復を示すリグレッションカバレッジを追加します。

索引モデルが issue の本文から書いたものです。

評価

技術スタック
python
領域
backend
issue の種類
バグ
難易度
4/5
見積もり時間
3〜5日
活発さ
活発
明瞭さ
明確に書かれている
初心者へのやさしさ
72/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。