container.list_objects with prefix and the local driver is very slow when a lot of files are stored
- Lingua principale
- Python
- Stelle
- 2.1k
- Fork
- 931
- Merge medio
- 1g 2h
- PR unite (30g)
- 4
Descrizione
## Summary
I have a large storage that I'm using libcloud to manage - in this specific case it's around 500,000 objects, and I've noticed that listing object using the local driver and a prefix is very slow (took almost 2 minutes to return 90 objects).
## Detailed Information
In the local driver implementation when calling list_objects with a prefix - we firstly get all the objects and serialize them and just after that we are filtering by the prefix.
Found on python3.6.8 with libcloud 3.2.0 and ubuntu 16:04
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
The issue points to the local driver's container.list_objects path; start there and inspect how objects are serialized before prefix filtering. Reproduce the report with a large local object set and a prefix, then verify that matching objects are returned without the reported multi-minute delay.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- python
- Ambito
- cloud
- Tipo di issue
- Bug
- Difficoltà
- 3/5
- Tempo stimato
- 1-2 giorni
- Stato di attività
- Ferma
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 35/100