mhx / mhx/dwarfs

Feature request: Improve block management for uncompressed blocks to save memory and enhance deduplication

Open
#139 1 comment 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
C++
Stars
2.6k
Forks
92
PR merge metrics
No merged PRs in 30d

Description

I would like to propose optimizing block management for uncompressed blocks in DwarFS. As it currently stands, uncompressed blocks are treated the same way as compressed blocks, meaning they are still loaded into memory and read sequentially from the beginning of the block from disk. This approach can be inefficient, especially when there is frequent access to uncompressed blocks. By allowing random access to the block without reading everything before the segment we need, or even not loading the block into memory at all, we could potentially save a significant amount of private memory.

`mmap()` could potentially enable efficient random access to uncompressed blocks and possibly eliminate the need to manually load them into memory entirely.

This feature would also be beneficial for the `mkdwarfs` process. If uncompressed blocks do not occupy private memory, they would not need to be counted toward the `--max-lookback-blocks` (-B) quota. This approach could effectively enlarge the deduplication lookup window without increasing the memory footprint. This idea is orthogonal to the proposal in https://github.com/mhx/dwarfs/issues/138, and these two methods can be combined to further optimize the deduplication process. For uncompressed blocks, they can still extend with byte granularity since `mmap()` allows for cheap random access.

I hope this proposal makes sense and I look forward to hearing your thoughts on its feasibility.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names no source files or tests. Start by locating uncompressed-block handling and the mkdwarfs implementation of --max-lookback-blocks (-B), then investigate whether mmap() can provide the proposed access pattern. Done means uncompressed blocks support efficient random access without unnecessary private-memory use and are handled correctly in the deduplication lookback quota.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
operating-systems, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.