GoogleCloudPlatform / GoogleCloudPlatform/gsutil

`du` command does not lookup subdirectories when passing a GCS subdirectory path that ends with '/'

Open
#1,180 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
918
Forks
335
PR merge metrics
No merged PRs in 30d

Description

# summary
The `gsutil du` command does not lookup subdirectories recursively when the given path met the following conditions:
- The path is a subdirectory of a bucket ( `gs://foo-bucket/bar-dir/` ) , not a bucket root ( `gs//foo-bucket/` )
- **The path ends with slash `/`**
- The given subdirectory path( `gs://foo-bucket/bar-dir/` ) has subdirectories ( `gs://foo-bucket/bar-dir/baz-dir` ) and files ( `gs://foo-bucket/bar-dir/baz-dir/some_file.png` ) in them.
- the path is a GCS path(gs://), not a AWS S3 path(s3://)

In this case, `gsutil du gs://foo-bucket/bar-dir/` does not lookup `gs://foo-bucket/bar-dir/baz-dir/` subdirectory, and does not show `some_file.png`, while `gsutil du gs://foo-bucket/bar-dir` does show.

Is it intended behavior? I feel this "last `/` behavior" quite odd.
It seems to me that `du` command should always lookup all subdirectories, as Linux `du` does.

# reproduce

```bash
$ gsutill cp some_big_file.png gs://foo-bucket/test_du/test_du1/
$ gsutil ls -l -h gs://foo-bucket/test_du/test_du1/
0 B 2021-01-18T05:29:07Z gs://foo-bucket/test_du/test_du1/
87.69 KiB 2021-01-18T05:34:43Z gs://foo-bucket/test_du/test_du1/some_big_file.png
TOTAL: 2 objects, 89794 bytes (87.69 KiB)

$ gsutil du -s -h gs://foo-bucket/test_du
87.69 KiB gs://foo-bucket/test_du
$ gsutil du -s -h gs://foo-bucket/test_du/
0 B gs://foo-bucket/test_du
```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the discrepancy with the two `gsutil du -s -h` commands shown for paths with and without a trailing slash. Trace the `du` command's handling of GCS subdirectory paths and compare its recursive lookup behavior; done means the trailing-slash form reports the same nested objects and total as the form without the slash.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, python
Domain
cli, cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.