activeloopai / activeloopai/deeplake
S3: create() requires s3:DeleteObject for index.lock, so an append-only writer is impossible
- Linguagem predominante
- C++
- Estrelas
- 9.2k
- Forks
- 722
- Métricas de merge de PRs
- Nenhum PR com merge em 30d
Descrição
**Version:** 4.7.0 · Linux aarch64, Python 3.12 · S3 storage
## What happens
`deeplake.create()` requires `s3:DeleteObject`. It takes and then removes a lock object:
```
StorageAccessDenied: [S3] Access denied:
_deeplake_log/d49ee0d83edd7d49ac93c13f64e2d946/index.lock
User: ... is not authorized to perform: s3:DeleteObject on
//tables/fresh/_deeplake_log/.../index.lock
```
`open` + `append` + `commit` on an existing dataset need **no** delete at all — measured below. So the delete requirement is specific to dataset creation.
## Reproduce
Credential with `s3:GetObject`, `s3:PutObject`, `s3:ListBucket`, `s3:GetBucketLocation`, and KMS encrypt+decrypt. No `s3:DeleteObject`:
```python
import deeplake, deeplake.types as t
# 1. append to an existing dataset — WORKS
ds = deeplake.open(url, creds=creds)
ds.append([{"event_id": "e2", "text": "second"}]); ds.commit("x")
# rows: 1 | active commits: 2
# 2. create a new dataset — FAILS on s3:DeleteObject for index.lock
deeplake.create(url2, creds=creds, schema={"event_id": t.Text(), "text": t.Text()})
```
## Why it matters
An append-only writer that **cannot destroy data** is a posture people deliberately want: a process that adds rows and provably cannot remove them. Today that is achievable for `open`/`append` but not for `create`, so any credential that must create a dataset also carries the right to delete objects under the whole prefix.
For us this is an audit record written by a process running untrusted code. Being unable to grant "add, never remove" for the whole lifecycle is the difference between a record that can be tampered with and one that cannot.
## Ask
Any of these solves it:
1. let the lock expire by S3 lifecycle rather than being deleted;
2. tolerate a stale lock at create (it is a create-time mutex, and a leftover object is not corruption);
3. make lock cleanup best-effort, so an `AccessDenied` on the delete does not fail the create.
Guia de contribuição
Avaliação
Esta issue ainda não foi avaliada.