GoogleCloudPlatform / GoogleCloudPlatform/gsutil

gsutil cp hangs on many small files when running in parallel

Open
#464 30 comments 19 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
918
Forks
335
PR merge metrics
No merged PRs in 30d

Description

I have a GCS bucket with millions of small files in different folders. When I run:

```bash
$ gsutil -m cp -r gs://my-bucket .
```
The process will eventually hang before completion, sometimes after 5 minutes and sometimes after several hours. This seems to be is 100% reproducible. I'm using version 4.27 but this has happened in older versions as well. As a workaround have I to use:

```bash
$ gsutil cp -r gs://my-bucket .
```

which works but it takes several days to download everything so it's not optimal.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.