source-cooperative / source-cooperative/data.source.coop

Support a batched / recursive prefix-delete operation

Open
#173 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Rust
Stars
24
Forks
6
Avg merge
1h 32m
Merged PRs (30d)
1

Description

Summary

The proxy forwards per-object DELETE /{bucket}/{key} but does not
implement S3's multi-object delete (POST /{bucket}?delete, the DeleteObjects
verb). Requests to it 404 with NoSuchBucket: <account_id> because the proxy
routes by object key and has no bucket-root handler.

This forces clients that need to delete many objects (e.g. an entire prefix) to
issue one DELETE per key. For large stores — a Zarr dataset can have thousands
to millions of chunks — that is thousands of round-trips and a lot of proxy load.

Context: source.coop PR source-cooperative/source.coop#403 added a file-explorer
"delete prefix" button; it currently lists the prefix and deletes each key
individually (concurrency 8) as a workaround. Correct, but slow at scale.

What we'd like

A way to delete many objects in one request. Two shapes, in preference order:

  1. Scoped recursive prefix-delete (preferred for our use case): a single
    request "delete everything under prefix P". The proxy authorizes the prefix
    once against the caller's scope, then lists + deletes server-side.
    • Pros: one authz check; works uniformly across S3/GCS/Azure backends;
      collapses N requests → 1 for the client.
  2. Standard S3 DeleteObjects passthrough: implement the bucket-root
    ?delete endpoint so standard S3 tooling (rclone, boto3, aws-cli bulk delete)
    works against the proxy.
    • Pros: S3-tool compatibility — a reason that stands on its own, independent
      of our delete button.

Known complexity (why this isn't a trivial passthrough)

  • Per-key authorization. The proxy is the per-object authz enforcement
    point. DeleteObjects bundles up to 1000 arbitrary keys in the request
    body (not the URL path). To be safe the proxy must parse the body and
    authorize every key against the caller's scope, returning per-key errors —
    otherwise one request could mix keys across products/accounts. This is the main
    cost and likely why no handler exists today. (A scoped prefix-delete avoids
    this: one prefix check.)
  • Backend heterogeneity. Only S3 has a native ~1000-key batch delete. GCS and
    Azure Blob have different/limited batch primitives, so for those backends the
    proxy would fan out to per-object deletes server-side anyway — the batch
    speedup is fully realized only for S3-backed connections.
  • Response semantics. DeleteObjects returns 200 with a per-key result body
    (including partial failures) and honors Quiet mode; the proxy must construct
    that correctly.
  • Read-only / unowned connections must continue to be rejected (we don't own
    that data).

Suggested scope

If the goal is just to speed up our own prefix deletes, option (1) is simpler and
safer. Implement option (2) only if S3-tool compatibility is independently
wanted. Either way, per-key/per-prefix authorization is non-negotiable.

Workaround in place

source.coop deletes prefixes per-object client-side today (PR
source-cooperative/source.coop#403, S3UploadService.deletePrefix). No action is
blocked; this issue is about performance at scale.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the existing per-object DELETE route and the source-cooperative/source.coop#403 workaround, including S3UploadService.deletePrefix. Decide whether the scoped recursive prefix-delete or S3 DeleteObjects shape is the target, then inspect the backend adapters and authorization behavior. Done means authorized read-only and unowned connections remain rejected and the chosen operation handles backend differences and partial results correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
authorization, backend-api-design, cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.