bloomreach / bloomreach/s4cmd

Does recursive get work?

Open
#84 1 comment 1 reaction 0 assignees View on GitHub
info-needed
Dominant language
Python
Stars
1.4k
Forks
209
PR merge metrics
No merged PRs in 30d

Description

I'm simply trying to download a bunch of files from s3, in a recursive manner. I am using the common convention that the s3 objects and prefixes can be viewed as directories and files, using "/" as the separator.

the following works using s3cmd:
s3cmd get -rf s3://bucket_name/ ./dirname/

the following does not work with s4cmd, using the exact same source/target locations:
s4cmd get -rf s3://bucket_name/ ./dirname/

I get errors like:
[Runtime Failure] Download Failure: , Source: s3://bucket/subdir/.
[Runtime Failure] Download Failure: , Source: s3://bucket/subdir/subdir2/subdir3a/.
[Runtime Failure] Download Failure: , Source: s3://bucket/subdir/subdir2/.
[Runtime Failure] Download Failure: , Source: s3://bucket/subdir/subdir2/subdir3b/.
[OSError] 2: No such file or directory
[OSError] 2: No such file or directory
[OSError] 2: No such file or directory
....

Is not s4cmd supposed to be drop-in replacement for s3cmd for these types of things? Does anyone have any troubleshooting advice? Can you point me to a public-access bucket on s3 where I can reproduce a recursive download using s4cmd?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the reported `s4cmd get -rf` command against a bucket containing nested prefixes, comparing it with the working `s3cmd` command. Trace the recursive download entry point and the reported missing-directory failures; done means nested objects download successfully without the shown errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cli, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.