GCS High Performance Parallel Listing
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 398
- Forks
- 181
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 36
Description
Hello Martin,
I am currently a Google SWE intern, and I am working as a part of the GCS team.
We are thinking of optimizing the listing operation in GCSFS, and want to get your initial approval on this.
In particular, we are thinking of utilizing multiple processes (using the concurrency.future library) and an optional GCS Insights Service to speed up the listing operation. Currently, the listing operation takes about 300 seconds to list 1 million objects. With the optimization, the listing operation can speed up 10x using the same setup. We have already experimented internally.
The Insights Service will be an optional configuration that the client can pass into the GCSFS listing call, so everything should be backwards compatible.
Let me know your initial thoughts. I will be working on the codebase and should hopefully have a PR out in the next few weeks.
My email is open: hanseaston@google.com, if you ever have anything you wish to discuss offline.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names no files or tests. Start by reviewing the current GCSFS listing implementation and the proposed optional GCS Insights Service and concurrency.future approach; done should preserve backwards compatibility and demonstrate faster large-object listings.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- gcp, python
- Domain
- cloud, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100