container.list_objects with prefix and the local driver is very slow when a lot of files are stored
- Langage dominant
- Python
- Étoiles
- 2.1k
- Forks
- 931
- Merge moyen
- 1 j 2 h
- PR mergées (30 j)
- 4
Description
## Summary
I have a large storage that I'm using libcloud to manage - in this specific case it's around 500,000 objects, and I've noticed that listing object using the local driver and a prefix is very slow (took almost 2 minutes to return 90 objects).
## Detailed Information
In the local driver implementation when calling list_objects with a prefix - we firstly get all the objects and serialize them and just after that we are filtering by the prefix.
Found on python3.6.8 with libcloud 3.2.0 and ubuntu 16:04
Guide de contribution
Ouvrir le guide de contribution
Piste de recherche
The issue points to the local driver's container.list_objects path; start there and inspect how objects are serialized before prefix filtering. Reproduce the report with a large local object set and a prefix, then verify that matching objects are returned without the reported multi-minute delay.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- python
- Domaine
- cloud
- Type d'issue
- Bug
- Difficulté
- 3/5
- Temps estimé
- 1-2 jours
- Activité
- À l'abandon
- Clarté
- Plutôt claire
- Accessibilité débutants
- 35/100