apache / apache/libcloud

container.list_objects with prefix and the local driver is very slow when a lot of files are stored

Ouverte
#1,583 2 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
stale
Langage dominant
Python
Étoiles
2.1k
Forks
931
Merge moyen
1 j 2 h
PR mergées (30 j)
4

Description

## Summary

I have a large storage that I'm using libcloud to manage - in this specific case it's around 500,000 objects, and I've noticed that listing object using the local driver and a prefix is very slow (took almost 2 minutes to return 90 objects).

## Detailed Information
In the local driver implementation when calling list_objects with a prefix - we firstly get all the objects and serialize them and just after that we are filtering by the prefix.

Found on python3.6.8 with libcloud 3.2.0 and ubuntu 16:04

Guide de contribution

Ouvrir le guide de contribution

Piste de recherche

The issue points to the local driver's container.list_objects path; start there and inspect how objects are serialized before prefix filtering. Reproduce the report with a large local object set and a prefix, then verify that matching objects are returned without the reported multi-minute delay.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
python
Domaine
cloud
Type d'issue
Bug
Difficulté
3/5
Temps estimé
1-2 jours
Activité
À l'abandon
Clarté
Plutôt claire
Accessibilité débutants
35/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.