borg2: allocate, copy / move, free less memory
Open
- Dominant language
- Python
- Stars
- 13.7k
- Forks
- 875
- Avg merge
- 11h 15m
- Merged PRs (30d)
- 192
Description
From a recent profiling:
The one large item that isn't crypto or I/O is memory management
memmove 17.1% + madvise 6.8% = ~24% of the critical path — copying each ~2 MB chunk between chunker, compressor and encrypter, then handing multi-MB buffers back to the OS.
Contributor guide
Research direction
Start with the profiling results in the issue and trace how roughly 2 MB chunks move between the chunker, compressor, and encrypter, including when multi-MB buffers are returned to the OS. Done means reducing the reported memmove and madvise share of the critical path without changing backup behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100