source-cooperative / source-cooperative/data.source.coop
Support a batched / recursive prefix-delete operation
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 24
- Forks
- 6
- Avg merge
- 1h 32m
- Merged PRs (30d)
- 1
Description
Summary
The proxy forwards per-object DELETE /{bucket}/{key} but does not
implement S3's multi-object delete (POST /{bucket}?delete, the DeleteObjects
verb). Requests to it 404 with NoSuchBucket: <account_id> because the proxy
routes by object key and has no bucket-root handler.
This forces clients that need to delete many objects (e.g. an entire prefix) to
issue one DELETE per key. For large stores — a Zarr dataset can have thousands
to millions of chunks — that is thousands of round-trips and a lot of proxy load.
Context: source.coop PR source-cooperative/source.coop#403 added a file-explorer
"delete prefix" button; it currently lists the prefix and deletes each key
individually (concurrency 8) as a workaround. Correct, but slow at scale.
What we'd like
A way to delete many objects in one request. Two shapes, in preference order:
- Scoped recursive prefix-delete (preferred for our use case): a single
request "delete everything under prefixP". The proxy authorizes the prefix
once against the caller's scope, then lists + deletes server-side.- Pros: one authz check; works uniformly across S3/GCS/Azure backends;
collapses N requests → 1 for the client.
- Pros: one authz check; works uniformly across S3/GCS/Azure backends;
- Standard S3
DeleteObjectspassthrough: implement the bucket-root
?deleteendpoint so standard S3 tooling (rclone, boto3, aws-cli bulk delete)
works against the proxy.- Pros: S3-tool compatibility — a reason that stands on its own, independent
of our delete button.
- Pros: S3-tool compatibility — a reason that stands on its own, independent
Known complexity (why this isn't a trivial passthrough)
- Per-key authorization. The proxy is the per-object authz enforcement
point.DeleteObjectsbundles up to 1000 arbitrary keys in the request
body (not the URL path). To be safe the proxy must parse the body and
authorize every key against the caller's scope, returning per-key errors —
otherwise one request could mix keys across products/accounts. This is the main
cost and likely why no handler exists today. (A scoped prefix-delete avoids
this: one prefix check.) - Backend heterogeneity. Only S3 has a native ~1000-key batch delete. GCS and
Azure Blob have different/limited batch primitives, so for those backends the
proxy would fan out to per-object deletes server-side anyway — the batch
speedup is fully realized only for S3-backed connections. - Response semantics.
DeleteObjectsreturns 200 with a per-key result body
(including partial failures) and honorsQuietmode; the proxy must construct
that correctly. - Read-only / unowned connections must continue to be rejected (we don't own
that data).
Suggested scope
If the goal is just to speed up our own prefix deletes, option (1) is simpler and
safer. Implement option (2) only if S3-tool compatibility is independently
wanted. Either way, per-key/per-prefix authorization is non-negotiable.
Workaround in place
source.coop deletes prefixes per-object client-side today (PR
source-cooperative/source.coop#403, S3UploadService.deletePrefix). No action is
blocked; this issue is about performance at scale.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the existing per-object DELETE route and the source-cooperative/source.coop#403 workaround, including S3UploadService.deletePrefix. Decide whether the scoped recursive prefix-delete or S3 DeleteObjects shape is the target, then inspect the backend adapters and authorization behavior. Done means authorized read-only and unowned connections remain rejected and the chosen operation handles backend differences and partial results correctly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- authorization, backend-api-design, cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100