quickwit-oss / quickwit-oss/quickwit
Reduce part size on Google Storage / Opendal
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 11.7k
- Forks
- 597
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 37
Description
The opendal storage implementation, when chunking (multipart) is enabled requires gathering all data in RAM before upload. There is no stream there.
The RAM usage is hence inflated. The main effect was curbed due to #6492 but we would benefit from reducing the part size a notch. The trade off is PUT requests costs.
Bringing it down from 5GB to 1GB for instance would really help without hurting user's wallets to much.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the Rust OpenDAL storage implementation and trace the multipart chunking path and its current 5GB part-size setting. Change the default toward 1GB, then verify the relevant storage behavior and confirm that the resulting trade-off is reduced peak RAM usage with increased PUT requests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- cloud
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 62/100