apache / apache/libcloud

container.list_objects with prefix and the local driver is very slow when a lot of files are stored

Offen
#1,583 2 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
stale
Vorherrschende Sprache
Python
Sterne
2.1k
Forks
931
Ø Merge
1 T. 2 Std.
Gemergte PRs (30 T.)
4

Beschreibung

## Summary

I have a large storage that I'm using libcloud to manage - in this specific case it's around 500,000 objects, and I've noticed that listing object using the local driver and a prefix is very slow (took almost 2 minutes to return 90 objects).

## Detailed Information
In the local driver implementation when calling list_objects with a prefix - we firstly get all the objects and serialize them and just after that we are filtering by the prefix.

Found on python3.6.8 with libcloud 3.2.0 and ubuntu 16:04

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

The issue points to the local driver's container.list_objects path; start there and inspect how objects are serialized before prefix filtering. Reproduce the report with a large local object set and a prefix, then verify that matching objects are returned without the reported multi-minute delay.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
python
Bereich
cloud
Issue-Typ
Bug
Schwierigkeit
3/5
Geschätzter Aufwand
1-2 Tage
Aktivitätsstatus
Veraltet
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
35/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.