borgbackup / borgbackup/borgstore

Let backends declare thread_safe and allow concurrent backend calls in Store (follow-up to #206)

Open
#208 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
25
Forks
8
Avg merge
16m
Merged PRs (30d)
8

Description

Follow-up to #206 / #207 (from Claude, reviewed by TW).

#207 serializes all Store operations with one internal lock. That is correct and sufficient for today's callers (borg's #9988 pack store-thread issues one op at a time), but it means the store can never execute two backend calls concurrently - even on backends where that would be safe and beneficial.

## Idea

Let a backend declare itself safe for concurrent calls, and let `Store` exploit that:

1. **Declaration**: `BackendBase.thread_safe: bool = False` class attribute. posixfs can honestly set `True` (independent syscalls per call). sftp/rest/rclone/s3 stay `False` with their single session - or later become `True` via a connection pool.
2. **Store side**: this is *not* just "skip the lock" - the Store's own bookkeeping (`_stats` Counter updates, writethrough-cache accounting, `_cache_disabled`) needs protection regardless of the backend. The finer-grained design: keep a small lock for stats/cache mutations, but allow the backend calls themselves (`backend.load/store/delete/...`) to run outside the lock when `backend.thread_safe` is true.
3. **Cache backend**: the cache backend is a second, independent backend with its own `thread_safe` - concurrent primary + cache calls need both declarations (or per-backend locks).

## Why bother (later)

- multiple concurrent uploads to hide per-request latency on cloud backends (the "riskier follow-up" scoped out of borgbackup/borg#9988),
- parallel prefetch on extract,
- concurrent range-reads for `get_many` on packs living in different objects.

None of these exist as callers today, which is why #207 deliberately kept the simple unconditional lock. This issue exists so the design above does not get lost, and to be picked up once a real parallel-I/O caller shows up.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.