borg2: use less memory for the index
- Dominant language
- Python
- Stars
- 13.7k
- Forks
- 875
- Avg merge
- 11h 15m
- Merged PRs (30d)
- 192
Description
Currently, there is just 1 index and it is fully in-memory (RAM).
```
ChunkIndex: object_id -> flags, size, pack_id, obj_offset, obj_size # 32 + 4 + 4 + 32 + 4 + 4 = 80B
```
borg create: for deduplication against already existing objects, we only need to know which object_ids we already have in the repository (and maybe size and flags could be also useful):
```
DedupIndex: object_id -> flags, size # 32 + 4 + 4 = 40B
```
We would also have a ChunkIndex, but only for the new objects of that session. This would roughly reduce the RAM usage by 50% for a usual daily backup.
borg extract: to restore a backup, we just need the pack infos:
```
PackIndex: object_id -> pack_id, obj_offset, obj_size # 32 + 32 + 4 + 4 = 72B
```
Maybe not worth it, the ChunkIndex is only 10% more.
Contributor guide
Research direction
No files or tests are named. Start by locating the current ChunkIndex implementation and the index-loading paths used by borg create and borg extract, then measure their memory use. Done would require an agreed index design that reduces create-time RAM while preserving deduplication and restore behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100