fxstein / fxstein/GoProX

Feature: Delta patch compress optional files

Open
#26 1 comment 0 reactions 1 assignee Claimed by @fxstein View on GitHub
enhancement further investigation
Dominant language
Shell
Stars
27
Forks
4
PR merge metrics
No merged PRs in 30d

Description

The storage requirements for `goprox` are very significant when all layers of the library are being kept for future re-prosessing.

There are basically 3 copies of data: `archive`, `imported` and `processed`

Both `archive` and `imported` are optional lineage copies that allow a user to go back to the originals coming off a camera and re-process the entire workflow. With early versions of `goprox` this is desirable as logic changes and even bugfixes can easily be applied to the original media.

Since the delta between `imported` and `processed` media is metadata only - we are not resampling or recompressing the media files - delta patches could be leveraged to replace media file with significantly smaller delta files, while allowing to restore the original file later on.

Initial testing has resulted in a 99% reduction in storage required to hold `imported` delta patch files, compared to the original media files.

Example:

```
xdelta3 -S djw -s /Users/oratzes/goprox-test-moved/imported/2022/20221011/20221011151401_GoPro_Hero10_2442_G1541642.JPG /Users/oratzes/goprox-test-moved/processed/JPEG/2022/20221011/P_20221011151401_GoPro_Hero10_2442_G1541642.jpg P_20221011151401_GoPro_Hero10_2442_G1541642.xdelta
```

produces a 1.4kB delta file compared to the original 4.7MB image. This would allow users to keep the entire `imported` media path while consuming 99% less storage than a full copy of the data.

`goprox` needs to be able to generate the delta patch files, replace the original files in `imported` and restore them on demand as needed for future processing.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.