apple / apple/foundationdb

Use standard format for backup files

Open
#2,259 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

If backup files use a standard format, such as Protocol Buffers or FlatBuffers, digesting the backup will not need the verbose and error-prone deserialization process.

It will make life easier to inspect backup data using external tools, such as Spark.

Since we are going to have a new backup system, it worth investigating how hard it is to incorporate this into the new backup system and the new performant restore system.

[This is a offline discussion with @apkar.]

Contributor guide

Open the contributing guide

Research direction

Review the new backup system and performant restore system mentioned in the issue, then investigate Protocol Buffers and FlatBuffers as standard formats for backup data and external Spark inspection. Done means documenting whether and how one of these formats can be incorporated, including the effort and implications for backup and restore.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, spark
Domain
data-engineering, databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.