borgbackup / borgbackup/borg

feature request: hash-based exclusion

Open
#4,699 7 comments 0 reactions 1 assignee Claimed by @ThomasWaldmann View on GitHub
patterns
Dominant language
Python
Stars
13.7k
Forks
875
Avg merge
11h 15m
Merged PRs (30d)
192

Description

I would find it very convenient to be able to include in my --exclude-from file entries of the form "hash:94aa9aa957ac131a6cc61218c5b3128eec32077ced97ad13934876d0" (or similar). My main intended use of this feature is to run "borg recreate" on every archive in order to purge all copies from my archives of certain large files that in retrospect I should not have backed up.

Normally, hash-based exclusion would not be requested, in which case the existence of this feature would not slow down processing. A set of to-be-excluded hashes would be stored, and if it is non-empty, each file as it is processed would have its hash computed and checked for membership in the hashes-to-be-excluded set.

As an optimization, one could use "hash:(hex hash),(file size)" which would allow the expensive hash computation to be skipped on any file whose size does not match the size of any to-be-excluded-based-on-hash content.

Some background info: After using borg to deduplicate 20 years of my own .tar.gz backups and other archives, I wondered which files were dominating the space, after deduplication. I wrote python code to list --json-lines the contents of each archive, including hash for every file, which I then wrote to a sqlite3 database and processed. So then for instance I can print out the size and hash of each of the 1000 largest unique file contents, and for each such file, I can list of everywhere it occurs. A subsequent step will be to make a list of "big junk" hash values and then to rewrite all archives in which those files occur, explicitly excluding those files. My current plan is to do this by matching full file paths as stored in the archive, but hash-based exclusion would be simpler and would more directly match my aim: to exclude certain files based on their content.

I had been using sha224, but Thomas suggests that blake2b-256 would be much faster. I don't care which hash is used as long as it offers negligible collision rate, though it is sometimes convenient to use a hash that widespread unix command-line utilities know how to calculate.

====

## Have you checked borgbackup docs, FAQ, and open Github issues?

Yes, frequently.

## Is this a BUG / ISSUE report or a QUESTION?

Feature request.

## System information. For client/server mode post info for both machines.

Intel Core i7 3.4GHz, 24GB SDRAM. Intel Core i5 1.6 GHz, 8GB SDRAM.

#### Your borg version (borg -V).

borg 1.1.5.

#### Operating system (distribution) and version.

Ubuntu 18.04 LTS, MacOS 10.14.

#### Hardware / network configuration, and filesystems used.

ext4, btrfs, hfs+, apfs.

#### How much data is handled by borg?

About 300GB after compression and deduplication.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.