apple / apple/foundationdb

Make backup size metrics useful again.

Open
#1,837 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

BackupConfig contains two property methods for storing the size of backup data written so far. They are rangeBytesWritten() and logBytesWritten(), and they are reported in backup layer status for each tag as `range_bytes_written` and `mutation_log_bytes_written`.

Currently, they are never cleared and just increase forever. This was previously useful to indicate how much backup data would have to be read in order to perform a restore as backups only had a single snapshot followed by some amount of logs.

In the continuous backup model now employed, which performs slow snapshots over time in series forever, these numbers are now only useful in showing the total amount of data written to the backup, which is not the same amount of data needed to restore to the most recent point in time. While it is probably still a good idea to track total bytes written, the bytes needed to restore should also be available as it was previously.

My proposed solution is to leave the current metrics as-is to count cumulative totals, but then to also add two more metrics: one for the size of the most recent complete snapshot, which would be used in the event of a restore to the latest version, and one for the amount of mutation log bytes needed to restore to the latest version. These could perhaps be reported in status as `restore_latest_range_bytes_needed` and `restore_latest_log_bytes_needed`.

I think the easiest way to implement this is as follows. BackupConfig needs four new properties:

1. latestSnapshotLogBytes
2. latestSnapshotRangeBytes
3. snapshotBeginLogBytes
4. snapshotBeginRangeBytes

And the logic is:

- When a snapshot begins, the current values of rangeBytesWritten and logBytesWritten are copied to snapshotBeginRangeBytes and snapshotBeginLogBytes, respectively.
- When a snapshot ends, the current values of snapshotBeginRangeBytes and snapshotBeginLogBytes are copied to latestSnapshotRangeBytes and latestSnapshotLogBytes, respectively.
- When generating status,
```
restore_latest_range_bytes_needed = rangeBytesWritten - latestSnapshotRangeBytes
restore_latest_log_bytes_needed = logBytesWritten - latestSnapshotLogBytes
```

Contributor guide

Open the contributing guide

Research direction

Locate BackupConfig and the backup layer status generation, then trace where snapshots begin and end. Add the four named properties and verify that status reports restore_latest_range_bytes_needed and restore_latest_log_bytes_needed as the specified cumulative differences.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.