borgbackup / borgbackup/borg

borg create: try to speed up unchanged file processing

Open
#8,552 2 comments 0 reactions 0 assignees View on GitHub
cmd: create
Dominant language
Python
Stars
13.7k
Forks
875
Avg merge
11h 15m
Merged PRs (30d)
192

Description

Early borg used to work filepath-based to deal with the source files (stat, open, read, xattrs, acls, fsflags).

That was problematic due to race conditions and thus was changed in borg 1.2 to use `open()` to get a file descriptor and then work with the fd everywhere possible, so we can be sure to always deal with the same fs object independently of its path.

But: fs api calls and especially `open()` can be rather slow for some filesystems, like network filesystems.

So, for an **unchanged** file (files cache hit), it currently does:
- st1 = stat(path)
- fd = open(path)
- st2 = fstat(fd)
- check st1 against st2
- here it notices (by checking against the files cache contents) that the file is unchanged and decides not to read the files content, but reuse the chunkids from files cache.
- reads xattrs, except when --noxattrs is given
- reads acls, except when --noacls is given
- reads fsflags (bsdflags), except when --noflags is given
- creates a new archive item

Review the code if it can be modified for the unchanged file case, so that the `open` and `fstat` call is not needed, without causing issues like re-introducing races.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.