bloomreach / bloomreach/s4cmd

s4cmd ls dies of out of memory on very large bucket

Open
#57 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.4k
Forks
209
PR merge metrics
No merged PRs in 30d

Description

I'm trying to do an ls on a big bucket (billions of objects) and i expected the ls to be "streaming". I guess it is first consuming all elements into the ram and only then spits to disk which won't work for ridiculously large buckets..

Linux killed it after 90 minutes of running the script, at that time it took about 25GB of RAM.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the s4cmd ls command entry point and reproduce the report against a sufficiently large bucket if available. Trace how object listings are collected and emitted; done means ls can process very large buckets without retaining the entire listing in memory and no longer dies from out-of-memory use.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.