activeloopai / activeloopai/deeplake

S3: create() requires s3:DeleteObject for index.lock, so an append-only writer is impossible

Đang mở
#3,162 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
C++
Star
9.2k
Fork
722
Chỉ số merge pull request
Không có pull request nào được merge trong 30 ngày

Mô tả

**Version:** 4.7.0 · Linux aarch64, Python 3.12 · S3 storage

## What happens

`deeplake.create()` requires `s3:DeleteObject`. It takes and then removes a lock object:

```
StorageAccessDenied: [S3] Access denied:
_deeplake_log/d49ee0d83edd7d49ac93c13f64e2d946/index.lock
User: ... is not authorized to perform: s3:DeleteObject on
//tables/fresh/_deeplake_log/.../index.lock
```

`open` + `append` + `commit` on an existing dataset need **no** delete at all — measured below. So the delete requirement is specific to dataset creation.

## Reproduce

Credential with `s3:GetObject`, `s3:PutObject`, `s3:ListBucket`, `s3:GetBucketLocation`, and KMS encrypt+decrypt. No `s3:DeleteObject`:

```python
import deeplake, deeplake.types as t

# 1. append to an existing dataset — WORKS
ds = deeplake.open(url, creds=creds)
ds.append([{"event_id": "e2", "text": "second"}]); ds.commit("x")
# rows: 1 | active commits: 2

# 2. create a new dataset — FAILS on s3:DeleteObject for index.lock
deeplake.create(url2, creds=creds, schema={"event_id": t.Text(), "text": t.Text()})
```

## Why it matters

An append-only writer that **cannot destroy data** is a posture people deliberately want: a process that adds rows and provably cannot remove them. Today that is achievable for `open`/`append` but not for `create`, so any credential that must create a dataset also carries the right to delete objects under the whole prefix.

For us this is an audit record written by a process running untrusted code. Being unable to grant "add, never remove" for the whole lifecycle is the difference between a record that can be tampered with and one that cannot.

## Ask

Any of these solves it:

1. let the lock expire by S3 lifecycle rather than being deleted;
2. tolerate a stale lock at create (it is a create-time mutex, and a leftover object is not corruption);
3. make lock cleanup best-effort, so an `AccessDenied` on the delete does not fail the create.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.