`aws s3 rm` should use batch deletes
- Dominant language
- Python
- Stars
- 17.3k
- Forks
- 4.6k
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 13
Description
When running the command `aws s3 rm --recursive s3://bucketname/path/`, I expect it to use batch object deletion to delete the files quickly with the fewest requests. It appears to be deleting files one at a time.
```
$ aws --version
aws-cli/1.11.13 Python/3.5.2 Linux/4.4.0-1041-aws botocore/1.4.70
```
To test:
Create one bucket with several files. Then sync that bucket to another bucket (this is the bucket we will delete everything from)
```aws s3 sync s3://source-bucket/ s3://bucket-to-delete/```
Then delete the contents of `bucket-to-delete`:
`aws s3 rm --recursive s3://bucket-to-delete/`
Notice that it lists each file to delete sequentially.
Re-sync from `source-bucket` to `bucket-to-delete` and re-delete things with `--debug` to get lots of detail. Save detail to `/tmp/out`
`aws s3 rm --debug --recursive s3://bucket-to-delete/ 2>&1 |tee /tmp/out`
Then inspect /tmp/out for HTTP requests to confirm the DELETE method was called multiple times, one for each object:
```
$ grep HTTP /tmp/out
2018-02-22 20:22:51,694 - MainThread - botocore.auth - DEBUG - HTTP request method: GET
2018-02-22 20:22:52,065 - Thread-4 - botocore.auth - DEBUG - HTTP request method: DELETE
2018-02-22 20:22:52,227 - Thread-6 - botocore.auth - DEBUG - HTTP request method: DELETE
2018-02-22 20:22:52,248 - Thread-8 - botocore.auth - DEBUG - HTTP request method: DELETE
2018-02-22 20:22:52,329 - Thread-9 - botocore.auth - DEBUG - HTTP request method: DELETE
2018-02-22 20:22:52,390 - Thread-4 - botocore.auth - DEBUG - HTTP request method: DELETE
2018-02-22 20:22:52,410 - Thread-11 - botocore.auth - DEBUG - HTTP request method: DELETE
2018-02-22 20:22:52,553 - Thread-4 - botocore.auth - DEBUG - HTTP request method: DELETE
```
Deleting portions of large buckets could be done much faster, with much less network overhead if the CLI tool would use batch deletes according to https://docs.aws.amazon.com/AmazonS3/latest/API/multiobjectdeleteapi.html
Contributor guide
Assessment
This issue has not been assessed yet.