bloomreach / bloomreach/s4cmd

[OSError] 2: No such file or directory

Open
#18 17 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.4k
Forks
209
PR merge metrics
No merged PRs in 30d

Description

I find when I try to download multiple Gigabytes from S3 to my local filesystem, this error will stop the process and therefore not download the rest of the files.

[OSError] 2: No such file or directory
[OSError] 2: No such file or directory
[OSError] 2: No such file or directory
[Thread Failure] [Errno 2] No such file or directory: './my_directory/my_file-thl8fgmwbcnxy36.tmp'

It usually succeeds in downloading a few hundred of ~1,500 files before it errors out, so I am left with let's say about half the directory's actual contents and lacking the rest of the files. I have found I can re-run the command multiple times and eventually it will finish. So it may be an edge case that crops up every few hundred or so transfers.

Here's the command:
`s4cmd.py get -r -s s3://my-s3-bucket/my_directory`

Any idea what's going on? I use s4cmd just about every day and absolutely love it; with this issue resolved I have nothing but praise for it!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing `s4cmd.py get -r -s s3://my-s3-bucket/my_directory` against a large set of S3 objects and inspect how recursive downloads create temporary local files and handle thread failures. The fix is done when the full download completes without `[Errno 2]` failures and no files are missing.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cli, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.