s4cmd ls dies of out of memory on very large bucket
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 209
- PR merge metrics
- No merged PRs in 30d
Description
I'm trying to do an ls on a big bucket (billions of objects) and i expected the ls to be "streaming". I guess it is first consuming all elements into the ram and only then spits to disk which won't work for ridiculously large buckets..
Linux killed it after 90 minutes of running the script, at that time it took about 25GB of RAM.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the s4cmd ls command entry point and reproduce the report against a sufficiently large bucket if available. Trace how object listings are collected and emitted; done means ls can process very large buckets without retaining the entire listing in memory and no longer dies from out-of-memory use.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100