Use standard format for backup files
- Dominant language
- C++
- Stars
- 16.7k
- Forks
- 1.6k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 126
Description
If backup files use a standard format, such as Protocol Buffers or FlatBuffers, digesting the backup will not need the verbose and error-prone deserialization process.
It will make life easier to inspect backup data using external tools, such as Spark.
Since we are going to have a new backup system, it worth investigating how hard it is to incorporate this into the new backup system and the new performant restore system.
[This is a offline discussion with @apkar.]
Contributor guide
Research direction
Review the new backup system and performant restore system mentioned in the issue, then investigate Protocol Buffers and FlatBuffers as standard formats for backup data and external Spark inspection. Done means documenting whether and how one of these formats can be incorporated, including the effort and implications for backup and restore.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, spark
- Domain
- data-engineering, databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100