[OSError] 2: No such file or directory
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 209
- PR merge metrics
- No merged PRs in 30d
Description
I find when I try to download multiple Gigabytes from S3 to my local filesystem, this error will stop the process and therefore not download the rest of the files.
[OSError] 2: No such file or directory
[OSError] 2: No such file or directory
[OSError] 2: No such file or directory
[Thread Failure] [Errno 2] No such file or directory: './my_directory/my_file-thl8fgmwbcnxy36.tmp'
It usually succeeds in downloading a few hundred of ~1,500 files before it errors out, so I am left with let's say about half the directory's actual contents and lacking the rest of the files. I have found I can re-run the command multiple times and eventually it will finish. So it may be an edge case that crops up every few hundred or so transfers.
Here's the command:
`s4cmd.py get -r -s s3://my-s3-bucket/my_directory`
Any idea what's going on? I use s4cmd just about every day and absolutely love it; with this issue resolved I have nothing but praise for it!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing `s4cmd.py get -r -s s3://my-s3-bucket/my_directory` against a large set of S3 objects and inspect how recursive downloads create temporary local files and handle thread failures. The fix is done when the full download completes without `[Errno 2]` failures and no files are missing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cli, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100