container.list_objects with prefix and the local driver is very slow when a lot of files are stored
- Dominant language
- Python
- Stars
- 2.1k
- Forks
- 931
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 4
Description
## Summary
I have a large storage that I'm using libcloud to manage - in this specific case it's around 500,000 objects, and I've noticed that listing object using the local driver and a prefix is very slow (took almost 2 minutes to return 90 objects).
## Detailed Information
In the local driver implementation when calling list_objects with a prefix - we firstly get all the objects and serialize them and just after that we are filtering by the prefix.
Found on python3.6.8 with libcloud 3.2.0 and ubuntu 16:04
Contributor guide
Research direction
The issue points to the local driver's container.list_objects path; start there and inspect how objects are serialized before prefix filtering. Reproduce the report with a large local object set and a prefix, then verify that matching objects are returned without the reported multi-minute delay.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100