aws / aws/aws-cli

aws s3 cp doesn't work with prefix

Open
#1,454 48 comments 39 reactions 0 assignees View on GitHub
community feature-request implementation p2 s3 s3filters
Dominant language
Python
Stars
17.3k
Forks
4.6k
Avg merge
1d 2h
Merged PRs (30d)
13

Description

From a bucket containing millions of files, I want do download a few thousand based on their prefix.

Note: using --exclude "*" --include "myprefix" works, but is impractical since it lists all the files and applies the filter afterwards.

The problem is aws s3 cp adds a trailing "/" after the prefix I specify. In the example below you can see that logs/2015-08-17 becomes logs/2015-08-17/
This additional "/" leads to an empty result set, since the keys in the bucket have the form logs/YYYY-MM-DD-HH-MM-SS-RANDOM).

> aws s3 cp s3://mybucket/logs/2015-08-17 . --recursive --dryrun --debug

2015-08-17 21:48:32,026 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/1.7.27 Python/2.7.6 Linux/3.14.13-c9, botocore version: 0.108.0
2015-08-17 21:48:32,026 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['s3', 'cp', 's3://mybucket/logs/2015-08-17', '.', '--recursive', '--dryrun', '--debug']
....
2015-08-17 21:48:32,088 - MainThread - botocore.endpoint - DEBUG - Making request for (verify_ssl=True) with params: {'query_string': {u'prefix': u'logs/2015-08-17/', u'encoding-type': 'url'}, 'headers': {}, 'url_path': u'/mybucket', 'body': '', 'method': u'GET'}

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the documented `aws s3 cp` recursive dry run with `--debug` and inspect how the S3 prefix is formed. Done means a prefix such as `logs/2015-08-17` is queried without an added trailing slash and matching keys are returned.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cli, cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.