langgenius / langgenius/dify

V2 archive deletion does not verify marker state after an ambiguous DeleteObject failure

Open
#41,620 0 comments 1 reaction 1 assignee Claimed by @zhaohao1004 View on GitHub
🐞 bug cloud
Dominant language
TypeScript
Stars
156k
Forks
24.6k
Avg merge
22h 9m
Merged PRs (30d)
610

Description

### Self Checks

- [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542).
- [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general).
- [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones.
- [x] I confirm that I am using English to submit this report, otherwise it will be closed.
- [x] 【中文用户 & Non English User】请使用英语提交,否则会被关闭 :)
- [x] Please do not modify this template :) and fill in all the required fields.

### Dify version

Source-verified on `main` at `8707d0c82f6f8e812b5fb5b9202fdce016c80d09`.

### Cloud or Self Hosted

Cloud

### Steps to reproduce

V2 workflow archive deletion uses object-store markers to make delete/restore transitions idempotent. Marker cleanup currently checks whether a marker exists and then calls `DeleteObject`, but it does not verify the marker's state after the delete call.

1. Start with a V2 bundle that has a transition marker such as `_DELETE_STARTED`.
2. Run non-dry-run archive deletion or retry an already committed bundle that has `_DELETED` plus a stale `_DELETE_STARTED` marker.
3. Let the object store complete `DeleteObject`, but make the client lose or time out waiting for the response.
4. Observe that `ArchiveStorage.delete_object()` raises even though the marker may already be absent remotely.

The bundle is then reported as failed because `_delete_marker()` trusts the uncertain client outcome instead of checking the post-delete object state.

#### Root cause

[`WorkflowRunBundleArchiveMaintenance._delete_marker()`](https://github.com/langgenius/dify/blob/8707d0c82f6f8e812b5fb5b9202fdce016c80d09/api/services/retention/workflow_run/bundle_archive_maintenance.py#L1208-L1212) performs a pre-delete existence check followed by one delete call:

```python
if storage.object_exists(marker_key):
storage.delete_object(marker_key)
```

A transport timeout cannot determine whether the remote object store applied the delete. The implementation does not perform a post-delete `HEAD`, so an already-successful remote delete is treated as a failure.

#### Impact

- A successfully deleted marker can still fail the bundle and stop the current shard.
- An already committed delete with `_DELETED` and a stale transition marker may remain blocked during idempotent recovery.
- The current fail-closed behavior avoids unsafe forward progress, but cannot reconcile an ambiguous object-store response from actual state.

### ✔️ Expected Behavior

Use the post-delete marker state as the success criterion while retaining bounded retries and fail-closed behavior:

1. Change only `_delete_marker`; do not alter global S3/R2 or botocore retry configuration.
2. After a successful `DeleteObject`, or after a retryable delete error, perform `HEAD` through `object_exists()`.
3. Return success only when `HEAD` authoritatively confirms the marker is absent.
4. If the marker still exists, retry with finite backoff.
5. If `HEAD` fails, a non-retryable delete error occurs, or retries are exhausted, fail closed.

Regression coverage should include:

- Delete raises a retryable error, but `HEAD` confirms the marker is already absent.
- The first delete fails, the marker remains, and a retry succeeds.
- The marker remains after all attempts.
- `HEAD` cannot determine marker state.
- A bundle with `_DELETED` and only a stale `_DELETE_STARTED` marker recovers idempotently.
- A non-retryable delete error is not retried.

### ❌ Actual Behavior

`_delete_marker()` performs no post-delete state check. A retryable transport error is returned as a bundle failure even when the remote delete may already have succeeded, while a nominally successful delete is also not verified.

No archive Parquet objects are deleted by this path; the affected objects are transition marker files. Fail-closed behavior should remain in place whenever absence cannot be confirmed.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.