borgbackup / borgbackup/borgstore
Let backends declare thread_safe and allow concurrent backend calls in Store (follow-up to #206)
- Dominant language
- Python
- Stars
- 25
- Forks
- 8
- Avg merge
- 16m
- Merged PRs (30d)
- 8
Description
Follow-up to #206 / #207 (from Claude, reviewed by TW).
#207 serializes all Store operations with one internal lock. That is correct and sufficient for today's callers (borg's #9988 pack store-thread issues one op at a time), but it means the store can never execute two backend calls concurrently - even on backends where that would be safe and beneficial.
## Idea
Let a backend declare itself safe for concurrent calls, and let `Store` exploit that:
1. **Declaration**: `BackendBase.thread_safe: bool = False` class attribute. posixfs can honestly set `True` (independent syscalls per call). sftp/rest/rclone/s3 stay `False` with their single session - or later become `True` via a connection pool.
2. **Store side**: this is *not* just "skip the lock" - the Store's own bookkeeping (`_stats` Counter updates, writethrough-cache accounting, `_cache_disabled`) needs protection regardless of the backend. The finer-grained design: keep a small lock for stats/cache mutations, but allow the backend calls themselves (`backend.load/store/delete/...`) to run outside the lock when `backend.thread_safe` is true.
3. **Cache backend**: the cache backend is a second, independent backend with its own `thread_safe` - concurrent primary + cache calls need both declarations (or per-backend locks).
## Why bother (later)
- multiple concurrent uploads to hide per-request latency on cloud backends (the "riskier follow-up" scoped out of borgbackup/borg#9988),
- parallel prefetch on extract,
- concurrent range-reads for `get_many` on packs living in different objects.
None of these exist as callers today, which is why #207 deliberately kept the simple unconditional lock. This issue exists so the design above does not get lost, and to be picked up once a real parallel-I/O caller shows up.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.