aws s3 cp doesn't work with prefix
- Dominant language
- Python
- Stars
- 17.3k
- Forks
- 4.6k
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 13
Description
From a bucket containing millions of files, I want do download a few thousand based on their prefix.
Note: using --exclude "*" --include "myprefix" works, but is impractical since it lists all the files and applies the filter afterwards.
The problem is aws s3 cp adds a trailing "/" after the prefix I specify. In the example below you can see that logs/2015-08-17 becomes logs/2015-08-17/
This additional "/" leads to an empty result set, since the keys in the bucket have the form logs/YYYY-MM-DD-HH-MM-SS-RANDOM).
> aws s3 cp s3://mybucket/logs/2015-08-17 . --recursive --dryrun --debug
2015-08-17 21:48:32,026 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/1.7.27 Python/2.7.6 Linux/3.14.13-c9, botocore version: 0.108.0
2015-08-17 21:48:32,026 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['s3', 'cp', 's3://mybucket/logs/2015-08-17', '.', '--recursive', '--dryrun', '--debug']
....
2015-08-17 21:48:32,088 - MainThread - botocore.endpoint - DEBUG - Making request for (verify_ssl=True) with params: {'query_string': {u'prefix': u'logs/2015-08-17/', u'encoding-type': 'url'}, 'headers': {}, 'url_path': u'/mybucket', 'body': '', 'method': u'GET'}
Contributor guide
Research direction
Start by reproducing the documented `aws s3 cp` recursive dry run with `--debug` and inspect how the S3 prefix is formed. Done means a prefix such as `logs/2015-08-17` is queried without an added trailing slash and matching keys are returned.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cli, cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100