GoogleCloudPlatform / GoogleCloudPlatform/gsutil
gsutil cp hangs on many small files when running in parallel
Open
- Dominant language
- Python
- Stars
- 918
- Forks
- 335
- PR merge metrics
- No merged PRs in 30d
Description
I have a GCS bucket with millions of small files in different folders. When I run:
```bash
$ gsutil -m cp -r gs://my-bucket .
```
The process will eventually hang before completion, sometimes after 5 minutes and sometimes after several hours. This seems to be is 100% reproducible. I'm using version 4.27 but this has happened in older versions as well. As a workaround have I to use:
```bash
$ gsutil cp -r gs://my-bucket .
```
which works but it takes several days to download everything so it's not optimal.
Contributor guide
Assessment
This issue has not been assessed yet.