borgbackup / borgbackup/borg

borg2: use less memory for the index

Open
#9,818 1 comment 0 reactions 0 assignees View on GitHub
c: index or hashtable cmd: create cmd: extract
Dominant language
Python
Stars
13.7k
Forks
875
Avg merge
11h 15m
Merged PRs (30d)
192

Description

Currently, there is just 1 index and it is fully in-memory (RAM).

```
ChunkIndex: object_id -> flags, size, pack_id, obj_offset, obj_size # 32 + 4 + 4 + 32 + 4 + 4 = 80B
```

borg create: for deduplication against already existing objects, we only need to know which object_ids we already have in the repository (and maybe size and flags could be also useful):
```
DedupIndex: object_id -> flags, size # 32 + 4 + 4 = 40B
```
We would also have a ChunkIndex, but only for the new objects of that session. This would roughly reduce the RAM usage by 50% for a usual daily backup.

borg extract: to restore a backup, we just need the pack infos:
```
PackIndex: object_id -> pack_id, obj_offset, obj_size # 32 + 32 + 4 + 4 = 72B
```
Maybe not worth it, the ChunkIndex is only 10% more.

Contributor guide

Open the contributing guide

Research direction

No files or tests are named. Start by locating the current ChunkIndex implementation and the index-loading paths used by borg create and borg extract, then measure their memory use. Done would require an agreed index design that reduces create-time RAM while preserving deduplication and restore behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.