activeloopai / activeloopai/deeplake

S3: create() requires s3:DeleteObject for index.lock, so an append-only writer is impossible

未关闭
#3,162 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
C++
星标
9.2k
派生
722
PR 合并指标
30 天内没有已合并 PR

描述

**Version:** 4.7.0 · Linux aarch64, Python 3.12 · S3 storage

## What happens

`deeplake.create()` requires `s3:DeleteObject`. It takes and then removes a lock object:

```
StorageAccessDenied: [S3] Access denied:
_deeplake_log/d49ee0d83edd7d49ac93c13f64e2d946/index.lock
User: ... is not authorized to perform: s3:DeleteObject on
//tables/fresh/_deeplake_log/.../index.lock
```

`open` + `append` + `commit` on an existing dataset need **no** delete at all — measured below. So the delete requirement is specific to dataset creation.

## Reproduce

Credential with `s3:GetObject`, `s3:PutObject`, `s3:ListBucket`, `s3:GetBucketLocation`, and KMS encrypt+decrypt. No `s3:DeleteObject`:

```python
import deeplake, deeplake.types as t

# 1. append to an existing dataset — WORKS
ds = deeplake.open(url, creds=creds)
ds.append([{"event_id": "e2", "text": "second"}]); ds.commit("x")
# rows: 1 | active commits: 2

# 2. create a new dataset — FAILS on s3:DeleteObject for index.lock
deeplake.create(url2, creds=creds, schema={"event_id": t.Text(), "text": t.Text()})
```

## Why it matters

An append-only writer that **cannot destroy data** is a posture people deliberately want: a process that adds rows and provably cannot remove them. Today that is achievable for `open`/`append` but not for `create`, so any credential that must create a dataset also carries the right to delete objects under the whole prefix.

For us this is an audit record written by a process running untrusted code. Being unable to grant "add, never remove" for the whole lifecycle is the difference between a record that can be tampered with and one that cannot.

## Ask

Any of these solves it:

1. let the lock expire by S3 lifecycle rather than being deleted;
2. tolerate a stale lock at create (it is a create-time mutex, and a leftover object is not corruption);
3. make lock cleanup best-effort, so an `AccessDenied` on the delete does not fail the create.

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。