borgbackup / borgbackup/borg

Add `benchmark crud-modes`

Open
#9,167 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
13.7k
Forks
875
Avg merge
11h 15m
Merged PRs (30d)
192

Description

/kind enhancement

Related: #9165 and #9166

Following the discussion in #9104 it's quite hard to properly benchmark Borg's various encryption modes right now. First, it's hard to benchmark multiple modes for comparison, because the user is required to manually create and later delete test repos using the designated encryption modes. Second, it's hard to use Borg's benchmark results to draw conclusions, see #9165 and #9166.

I thus like to suggest adding a new `benchmark crud-modes` command (name is open for debate, didn't find a better one…) that is basically a wrapper for `benchmark crud` to create and later delete test repos with different encryption modes. Besides making benchmarking easier, it also promotes actually comparing the performance of different encryption modes, and not deciding based on a feeling what might be best.

This command should be added to both Borg 1.4 and Borg 2. The command should benchmark all available encryption modes after another. AFAIK there's no benefit in benchmarking both `repokey` and `keyfile`, so Borg should just benchmark one. Basically, `benchmark crud-mode` should do the following (assuming Borg 1.4; with Borg 2 the commands and encryption modes are different):

```sh
export BORG_PASSPHRASE=mypassphrase
for e in none authenticated authenticated-blake2 repokey repokey-blake2; do
borg init -e $e ./$e

echo "$e $(printf '%.0s=' $(seq $(( 64 - ${#e} - 1 ))))"
mkdir ./bench_data
borg benchmark crud ./$e ./bench_data

borg delete ./$e
rmdir bench_data
done
```

The results shown by `benchmark crud-modes` are identical to that of `benchmark crud`, with the only exception being that the results are grouped by encryption mode. Additionally, it might be reasonable to also print a summary:

**Including a summary**

I'm not entirely sure yet whether `benchmark crud-modes` should also analyse the results similar to what I did manually in https://github.com/borgbackup/borg/issues/9104#issuecomment-3482694134 and https://github.com/borgbackup/borg/issues/9104#issuecomment-3493049066, but I tend to "yes". I don't necessarily mean drawing conclusions, that's a bit hard to implement, but e.g. calculating and printing averages for various commands and samples. For example, `benchmark crud-modes` could include a summary section that prints the averages over all samples per command and encryption mode. Such summary section could look like the following (using results from #9104 and including just two encryption modes). WDYT?

```
summary ========================================================
C-*-* 466.94 MB/s with none encryption (average of all samples: 1.87s)
R-*-* 376.14 MB/s with none encryption (average of all samples: 1.64s)
U-*-* 4,543.96 MB/s with none encryption (average of all samples: 0.41s)
D-*-* 28,203.73 MB/s with none encryption (average of all samples: 0.05s)
C-*-* 487.28 MB/s with authenticated-blake2 encryption (average of all samples: 1.67s)
R-*-* 495.45 MB/s with authenticated-blake2 encryption (average of all samples: 1.22s)
U-*-* 3,672.36 MB/s with authenticated-blake2 encryption (average of all samples: 0.44s)
D-*-* 10,657.19 MB/s with authenticated-blake2 encryption (average of all samples: 0.10s)
```

**Support of `--json-lines`**

Similar to the suggested `benchmark crud --json-lines` (see #9165), the new `benchmark crud-modes` command should also support the `--json-lines` option that yields the same output as `benchmark crud --json-lines`, but additionally includes an `encryption` key and includes the mode used in its `id` key. Here's an example line with `--json-lines`:

```json
{"id": "authenticated_C-Z-BIG", "encryption": "authenticated", "command": "create1", "sample": "Z-BIG", "sample_count": 10, "sample_size": 100000000, "sample_random": false, "time": 4.460000, "io": 224320000}
```

With `--json-lines` the summary section should be omitted.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.