borgbackup / borgbackup/borg

Print `borg create --list --dry-run` to STDOUT to allow combination with `--paths-from-stdin`

Open
#8,710 2 comments 0 reactions 0 assignees View on GitHub
cmd: create
Dominant language
Python
Stars
13.7k
Forks
875
Avg merge
11h 15m
Merged PRs (30d)
192

Description

Currently, borg seems to list files to STDERR, which prevents its use in UNIX pipes. Of course, I could redirect `2>&1` but it would also include error messages and password prompts. The primary purpose of `borg create --list --dry-run ...` is to list files, so one could argue that therefore this should go to STDOUT and not STDERR. Also, this allows to use the borg pattern matching, then processing these somehow and finally pipe them back to `borg create ... --paths-from-stdin`.

My use-case is the following: I am using the pcloud remote of rclone to back up my data. This is slow when many small files are involved. As a test I arbitrarily tried to upload the git repo of [opendataeditor](https://github.com/okfn/opendataeditor) and saw from `vnstat -l` that upload speed is only a few kbit/s. As it was noted that [multiple parallel borg2 processes can concurrently write to the same repo](https://github.com/borgbackup/borg/discussions/6958#discussioncomment-11806464), I did exactly that.

However, these processes tended to "clump together" uploading the same files at the same time, which is why I wanted to shuffle the file order so that the different upload processes do not interfere as much. This is where `borg create --list --dry-run ... | shuf ` would become handy.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.