aws / aws/aws-cli

`aws s3 rm` should use batch deletes

Open
#3,163 8 comments 10 reactions 0 assignees View on GitHub
enhancement feature-request p2 s3
Dominant language
Python
Stars
17.3k
Forks
4.6k
Avg merge
1d 2h
Merged PRs (30d)
13

Description

When running the command `aws s3 rm --recursive s3://bucketname/path/`, I expect it to use batch object deletion to delete the files quickly with the fewest requests. It appears to be deleting files one at a time.

```
$ aws --version
aws-cli/1.11.13 Python/3.5.2 Linux/4.4.0-1041-aws botocore/1.4.70
```

To test:
Create one bucket with several files. Then sync that bucket to another bucket (this is the bucket we will delete everything from)

```aws s3 sync s3://source-bucket/ s3://bucket-to-delete/```

Then delete the contents of `bucket-to-delete`:
`aws s3 rm --recursive s3://bucket-to-delete/`

Notice that it lists each file to delete sequentially.

Re-sync from `source-bucket` to `bucket-to-delete` and re-delete things with `--debug` to get lots of detail. Save detail to `/tmp/out`

`aws s3 rm --debug --recursive s3://bucket-to-delete/ 2>&1 |tee /tmp/out`

Then inspect /tmp/out for HTTP requests to confirm the DELETE method was called multiple times, one for each object:
```
$ grep HTTP /tmp/out
2018-02-22 20:22:51,694 - MainThread - botocore.auth - DEBUG - HTTP request method: GET
2018-02-22 20:22:52,065 - Thread-4 - botocore.auth - DEBUG - HTTP request method: DELETE
2018-02-22 20:22:52,227 - Thread-6 - botocore.auth - DEBUG - HTTP request method: DELETE
2018-02-22 20:22:52,248 - Thread-8 - botocore.auth - DEBUG - HTTP request method: DELETE
2018-02-22 20:22:52,329 - Thread-9 - botocore.auth - DEBUG - HTTP request method: DELETE
2018-02-22 20:22:52,390 - Thread-4 - botocore.auth - DEBUG - HTTP request method: DELETE
2018-02-22 20:22:52,410 - Thread-11 - botocore.auth - DEBUG - HTTP request method: DELETE
2018-02-22 20:22:52,553 - Thread-4 - botocore.auth - DEBUG - HTTP request method: DELETE
```

Deleting portions of large buckets could be done much faster, with much less network overhead if the CLI tool would use batch deletes according to https://docs.aws.amazon.com/AmazonS3/latest/API/multiobjectdeleteapi.html

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.