container.list_objects with prefix and the local driver is very slow when a lot of files are stored
- 主要言語
- Python
- スター
- 2.1k
- フォーク
- 931
- 平均マージ
- 1日 2時間
- マージ済み PR(30日)
- 4
説明
## Summary
I have a large storage that I'm using libcloud to manage - in this specific case it's around 500,000 objects, and I've noticed that listing object using the local driver and a prefix is very slow (took almost 2 minutes to return 90 objects).
## Detailed Information
In the local driver implementation when calling list_objects with a prefix - we firstly get all the objects and serialize them and just after that we are filtering by the prefix.
Found on python3.6.8 with libcloud 3.2.0 and ubuntu 16:04
コントリビューションガイド
調査の方向性
The issue points to the local driver's container.list_objects path; start there and inspect how objects are serialized before prefix filtering. Reproduce the report with a large local object set and a prefix, then verify that matching objects are returned without the reported multi-minute delay.
索引モデルが issue の本文から書いたものです。
評価
- 技術スタック
- python
- 領域
- cloud
- issue の種類
- バグ
- 難易度
- 3/5
- 見積もり時間
- 1〜2日
- 活発さ
- 停滞
- 明瞭さ
- おおむね明確
- 初心者へのやさしさ
- 35/100