GoogleCloudPlatform / GoogleCloudPlatform/gsutil

spurious empty folders uploaded for each composite parallel upload in hierarchical buckets

Open
#1,890 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
918
Forks
335
PR merge metrics
No merged PRs in 30d

Description

Hi,

Not sure if this is the venue for this kind of problems but I am facing a disconcerting situation.
I did not find any reference to this problem anywhere either...

Here is the context:
We migrated from a standard bucket to a hierarchical one (very simple conf, no rules/acl/deletion protection) and discovered that any upload using either gsutil or gcloud storage will left empty folders named like this for each composite upload:

when using gcloud storare:
`/gcloud/tmp/parallel_composite_uploads/see_gcloud_storage_cp_help_for_details`
or, in our case with gsutil:
`/8660532868/gsutil/tmp/parallel_composite_uploads/for_details_see/gsutil_help_cp`

This furiously looks like a warning message get through the upload process for some reason...

The workarounds we applied in emergency in our scripts is to change the gsutil calls with:

gsutil -m -o GSUtil:parallel_composite_upload_threshold=0 cp ...

For gcloud, the recommendation is to deactivate composite uploads entirely:

`gcloud config set storage/parallel_composite_upload_enabled False`

which is obviously not a solution as it will slow down the uploads for non-hierarchical buckets too.

This is a very annoying situation as these buckets are used by automated scripts and we discovered this after the fact, basically butchering our backups with tens of thousands of such empty folders.

I don't know if this is a bug, a conf problem or anything in between, but gsutil being apparently deprecated in favor of gcloud storage my confidence in the google's platform is down another notch.

Would be happy to hear any advice / confirmation/infomration of my findings.

`gsutil --version`: 5.35

```
gcloud --version
Google Cloud SDK 550.0.0
alpha 2025.12.12
beta 2025.12.12
bq 2.1.26
bundled-python3-unix 3.13.10
core 2025.12.12
gcloud-crc32c 1.0.0
gsutil 5.35
minikube 1.37.0
skaffold 2.17.0
```

```
lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 24.04.3 LTS
Release: 24.04
Codename: noble
```
Cheers on the eve of the year.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing a parallel composite upload to a hierarchical bucket with gsutil and gcloud storage, using the versions and commands in the report. Compare the results with composite uploads disabled and determine whether empty folders are created by either tool; done means identifying the responsible upload path and preventing those folders without disabling composite uploads generally.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, python
Domain
cli, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.