activeloopai / activeloopai/deeplake

S3: create() requires s3:DeleteObject for index.lock, so an append-only writer is impossible

オープン
#3,162 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
主要言語
C++
スター
9.2k
フォーク
722
PR マージ指標
30日以内にマージされた PR はありません

説明

**Version:** 4.7.0 · Linux aarch64, Python 3.12 · S3 storage

## What happens

`deeplake.create()` requires `s3:DeleteObject`. It takes and then removes a lock object:

```
StorageAccessDenied: [S3] Access denied:
_deeplake_log/d49ee0d83edd7d49ac93c13f64e2d946/index.lock
User: ... is not authorized to perform: s3:DeleteObject on
//tables/fresh/_deeplake_log/.../index.lock
```

`open` + `append` + `commit` on an existing dataset need **no** delete at all — measured below. So the delete requirement is specific to dataset creation.

## Reproduce

Credential with `s3:GetObject`, `s3:PutObject`, `s3:ListBucket`, `s3:GetBucketLocation`, and KMS encrypt+decrypt. No `s3:DeleteObject`:

```python
import deeplake, deeplake.types as t

# 1. append to an existing dataset — WORKS
ds = deeplake.open(url, creds=creds)
ds.append([{"event_id": "e2", "text": "second"}]); ds.commit("x")
# rows: 1 | active commits: 2

# 2. create a new dataset — FAILS on s3:DeleteObject for index.lock
deeplake.create(url2, creds=creds, schema={"event_id": t.Text(), "text": t.Text()})
```

## Why it matters

An append-only writer that **cannot destroy data** is a posture people deliberately want: a process that adds rows and provably cannot remove them. Today that is achievable for `open`/`append` but not for `create`, so any credential that must create a dataset also carries the right to delete objects under the whole prefix.

For us this is an audit record written by a process running untrusted code. Being unable to grant "add, never remove" for the whole lifecycle is the difference between a record that can be tampered with and one that cannot.

## Ask

Any of these solves it:

1. let the lock expire by S3 lifecycle rather than being deleted;
2. tolerate a stale lock at create (it is a create-time mutex, and a leftover object is not corruption);
3. make lock cleanup best-effort, so an `AccessDenied` on the delete does not fail the create.

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。