borgbackup / borgbackup/borg

borg2: allocate, copy / move, free less memory

Open
#10,058 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
13.7k
Forks
875
Avg merge
11h 15m
Merged PRs (30d)
192

Description

From a recent profiling:

The one large item that isn't crypto or I/O is memory management

memmove 17.1% + madvise 6.8% = ~24% of the critical path — copying each ~2 MB chunk between chunker, compressor and encrypter, then handing multi-MB buffers back to the OS.

Contributor guide

Open the contributing guide

Research direction

Start with the profiling results in the issue and trace how roughly 2 MB chunks move between the chunker, compressor, and encrypter, including when multi-MB buffers are returned to the OS. Done means reducing the reported memmove and madvise share of the critical path without changing backup behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.