hemilabs / hemilabs/dave

internal/dave: improve snapshot archive storage logic

Open
#7 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2
Forks
0
PR merge metrics
No merged PRs in 30d

Description

Currently, Dave uses a `Repository` to store snapshots. Internally, a Repository looks like this:
```go
// Repository is a storage implementation which stores snapshots.
type Repository interface {
Metadata(ctx context.Context) (*RepositoryMeta, error)
MetadataUpdate(ctx context.Context, meta *RepositoryMeta) error

SnapshotAdd(ctx context.Context, snapshot *Snapshot) error
SnapshotByID(ctx context.Context, id string) (*Snapshot, error)
SnapshotList(ctx context.Context) ([]*Snapshot, error)
SnapshotRemove(ctx context.Context, id string) error
}
```

A Snapshot consists of metadata and one or more archives. This means that we only add a snapshot once all archives are created.

Archives are stored on the local file system before being moved to the repository, requiring enough storage for:
1. The data being backed up (e.g. 1TB)
2. The clone of data being backed up (same as 1; 1TB)
3. All archives of the cloned data (same as 2, compressed, e.g. 0.8x)

In cases where we are making a snapshot of multiple **very large** directories, it may work better to upload snapshot archives individually to the repository as they are created, slightly reducing the storage required to use Dave.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.