drivendataorg / drivendataorg/cloudpathlib

Performance improvement for glob

Open
#513 3 comments 0 reactions 0 assignees View on GitHub
GCP
Dominant language
Python
Stars
628
Forks
88
Avg merge
17h 28m
Merged PRs (30d)
2

Description

Hey, I've been using the GSPath globbing capabilities to glob over a fairly large GCS bucket (couple of gbs) and have been noticing that it takes a lot longer to process compared to a `google-cloud-storage` implementation.

```
list_blobs(match_glob="**/version_1/**")
```

Furthermore, when having task manager open when performing a glob on the bucket I observe significantly higher network footprint (when using GSPath) in comparison to the `list_blobs` implementation.

My guess is that cloudpathlib may potentially be sending more network request than necessary (correct me if I'm wrong)

Any reasons why we don't just leverage the match_glob arg for `GSPath`'s glob capabilities?

GCloud SDK `list_blobs(match_glob="")` reference below:
https://github.com/googleapis/python-storage/blob/main/google/cloud/storage/bucket.py#L1407

a `GSPath("/path/to/folder/").glob("**/version_1/**)` to my belief can be translated to `list_blobs` by doing the following:
`list_blobs(prefix="/path/to/folder", match_glob="**/version_1/**")`

happy to submit something if you would like this change incorporated :)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.