V2 archive deletion does not verify marker state after an ambiguous DeleteObject failure
- Dominant language
- TypeScript
- Stars
- 156k
- Forks
- 24.6k
- Avg merge
- 22h 9m
- Merged PRs (30d)
- 610
Description
### Self Checks
- [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542).
- [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general).
- [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones.
- [x] I confirm that I am using English to submit this report, otherwise it will be closed.
- [x] 【中文用户 & Non English User】请使用英语提交,否则会被关闭 :)
- [x] Please do not modify this template :) and fill in all the required fields.
### Dify version
Source-verified on `main` at `8707d0c82f6f8e812b5fb5b9202fdce016c80d09`.
### Cloud or Self Hosted
Cloud
### Steps to reproduce
V2 workflow archive deletion uses object-store markers to make delete/restore transitions idempotent. Marker cleanup currently checks whether a marker exists and then calls `DeleteObject`, but it does not verify the marker's state after the delete call.
1. Start with a V2 bundle that has a transition marker such as `_DELETE_STARTED`.
2. Run non-dry-run archive deletion or retry an already committed bundle that has `_DELETED` plus a stale `_DELETE_STARTED` marker.
3. Let the object store complete `DeleteObject`, but make the client lose or time out waiting for the response.
4. Observe that `ArchiveStorage.delete_object()` raises even though the marker may already be absent remotely.
The bundle is then reported as failed because `_delete_marker()` trusts the uncertain client outcome instead of checking the post-delete object state.
#### Root cause
[`WorkflowRunBundleArchiveMaintenance._delete_marker()`](https://github.com/langgenius/dify/blob/8707d0c82f6f8e812b5fb5b9202fdce016c80d09/api/services/retention/workflow_run/bundle_archive_maintenance.py#L1208-L1212) performs a pre-delete existence check followed by one delete call:
```python
if storage.object_exists(marker_key):
storage.delete_object(marker_key)
```
A transport timeout cannot determine whether the remote object store applied the delete. The implementation does not perform a post-delete `HEAD`, so an already-successful remote delete is treated as a failure.
#### Impact
- A successfully deleted marker can still fail the bundle and stop the current shard.
- An already committed delete with `_DELETED` and a stale transition marker may remain blocked during idempotent recovery.
- The current fail-closed behavior avoids unsafe forward progress, but cannot reconcile an ambiguous object-store response from actual state.
### ✔️ Expected Behavior
Use the post-delete marker state as the success criterion while retaining bounded retries and fail-closed behavior:
1. Change only `_delete_marker`; do not alter global S3/R2 or botocore retry configuration.
2. After a successful `DeleteObject`, or after a retryable delete error, perform `HEAD` through `object_exists()`.
3. Return success only when `HEAD` authoritatively confirms the marker is absent.
4. If the marker still exists, retry with finite backoff.
5. If `HEAD` fails, a non-retryable delete error occurs, or retries are exhausted, fail closed.
Regression coverage should include:
- Delete raises a retryable error, but `HEAD` confirms the marker is already absent.
- The first delete fails, the marker remains, and a retry succeeds.
- The marker remains after all attempts.
- `HEAD` cannot determine marker state.
- A bundle with `_DELETED` and only a stale `_DELETE_STARTED` marker recovers idempotently.
- A non-retryable delete error is not retried.
### ❌ Actual Behavior
`_delete_marker()` performs no post-delete state check. A retryable transport error is returned as a bundle failure even when the remote delete may already have succeeded, while a nominally successful delete is also not verified.
No archive Parquet objects are deleted by this path; the affected objects are transition marker files. Fail-closed behavior should remain in place whenever absence cannot be confirmed.
Contributor guide
Assessment
This issue has not been assessed yet.