borgbackup / borgbackup/borg

1.1-maint vs master performance difference

Open
#4,498 38 comments 0 reactions 0 assignees View on GitHub
documentation
Dominant language
Python
Stars
13.7k
Forks
875
Avg merge
11h 15m
Merged PRs (30d)
192

Description

In #3955 the question came up, why `borg create` with master branch code is slower than with 1.1-maint branch code. The benchmarks I did there were archiving lots of relatively small files.

I did a source code based analysis, checking the "hot" code path that is taken when backing up known, unchanged, regular files (like when doing a 2nd backup right after a first one, without much changes).

Found this difference, in `process_file` (simplified pseudo code that is new in master):
```
fd = open(path)
st_fd = fstat(fd)
stat_update_check(st_path, st_fd)
```

These changes were done when switching borg master branch to work based on a FD (file descriptor) to avoid race conditions and potential security issues. See #4043.

For this, master code does it like this:
- `st_path = stat(path)` - before dispatching to file-type handler
- dispatch based on st_path (e.g. to process_file, process_fifo, process_symlink, ...)
- `fd = open(path)` - get an FD, so we can do other operations based on this FD. **new in master** (1.1-maint did not open unchanged files and did acquire all metadata based on the path)
- stat_update_check: determining `st_fd = fstat(fd)` to check for a race condition ("did we dispatch to correct file-type handler?") by comparing st_fd to st_path. **this call / this check is new in master**
- reading file contents (based on open fd, but not in the "hot" case, here we know contents have not changed)
- reading bsdflags (based on open fd on linux, based on st_fd on others)
- reading xattrs (based on open fd)
- reading ACLs (based on open fd)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.