GoogleCloudPlatform / GoogleCloudPlatform/gsutil
`du` command does not lookup subdirectories when passing a GCS subdirectory path that ends with '/'
- Dominant language
- Python
- Stars
- 918
- Forks
- 335
- PR merge metrics
- No merged PRs in 30d
Description
# summary
The `gsutil du` command does not lookup subdirectories recursively when the given path met the following conditions:
- The path is a subdirectory of a bucket ( `gs://foo-bucket/bar-dir/` ) , not a bucket root ( `gs//foo-bucket/` )
- **The path ends with slash `/`**
- The given subdirectory path( `gs://foo-bucket/bar-dir/` ) has subdirectories ( `gs://foo-bucket/bar-dir/baz-dir` ) and files ( `gs://foo-bucket/bar-dir/baz-dir/some_file.png` ) in them.
- the path is a GCS path(gs://), not a AWS S3 path(s3://)
In this case, `gsutil du gs://foo-bucket/bar-dir/` does not lookup `gs://foo-bucket/bar-dir/baz-dir/` subdirectory, and does not show `some_file.png`, while `gsutil du gs://foo-bucket/bar-dir` does show.
Is it intended behavior? I feel this "last `/` behavior" quite odd.
It seems to me that `du` command should always lookup all subdirectories, as Linux `du` does.
# reproduce
```bash
$ gsutill cp some_big_file.png gs://foo-bucket/test_du/test_du1/
$ gsutil ls -l -h gs://foo-bucket/test_du/test_du1/
0 B 2021-01-18T05:29:07Z gs://foo-bucket/test_du/test_du1/
87.69 KiB 2021-01-18T05:34:43Z gs://foo-bucket/test_du/test_du1/some_big_file.png
TOTAL: 2 objects, 89794 bytes (87.69 KiB)
$ gsutil du -s -h gs://foo-bucket/test_du
87.69 KiB gs://foo-bucket/test_du
$ gsutil du -s -h gs://foo-bucket/test_du/
0 B gs://foo-bucket/test_du
```
Contributor guide
Research direction
Start by reproducing the discrepancy with the two `gsutil du -s -h` commands shown for paths with and without a trailing slash. Trace the `du` command's handling of GCS subdirectory paths and compare its recursive lookup behavior; done means the trailing-slash form reports the same nested objects and total as the form without the slash.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, python
- Domain
- cli, cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100