Feature request: Improve block management for uncompressed blocks to save memory and enhance deduplication
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 2.6k
- Forks
- 92
- PR merge metrics
- No merged PRs in 30d
Description
I would like to propose optimizing block management for uncompressed blocks in DwarFS. As it currently stands, uncompressed blocks are treated the same way as compressed blocks, meaning they are still loaded into memory and read sequentially from the beginning of the block from disk. This approach can be inefficient, especially when there is frequent access to uncompressed blocks. By allowing random access to the block without reading everything before the segment we need, or even not loading the block into memory at all, we could potentially save a significant amount of private memory.
`mmap()` could potentially enable efficient random access to uncompressed blocks and possibly eliminate the need to manually load them into memory entirely.
This feature would also be beneficial for the `mkdwarfs` process. If uncompressed blocks do not occupy private memory, they would not need to be counted toward the `--max-lookback-blocks` (-B) quota. This approach could effectively enlarge the deduplication lookup window without increasing the memory footprint. This idea is orthogonal to the proposal in https://github.com/mhx/dwarfs/issues/138, and these two methods can be combined to further optimize the deduplication process. For uncompressed blocks, they can still extend with byte granularity since `mmap()` allows for cheap random access.
I hope this proposal makes sense and I look forward to hearing your thoughts on its feasibility.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names no source files or tests. Start by locating uncompressed-block handling and the mkdwarfs implementation of --max-lookback-blocks (-B), then investigate whether mmap() can provide the proposed access pattern. Done means uncompressed blocks support efficient random access without unnecessary private-memory use and are handled correctly in the deduplication lookback quota.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- operating-systems, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100