Redesign the single file format for DTFile
Open
@lidezhu is already working on this.
Since Apr 23, 2021.
- Dominant language
- C++
- Stars
- 1k
- Forks
- 423
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 24
Description
We propose a "single file mode" for DTFile to store the snapshot generated / snapshot files for ingesting. Storing data in a single file is easy for transferring data between nodes.
Single File Format
But there are some problems for current design and implementation:
- The file format is not efficient. For example, we need to open and read the same file four times to get
Footer,meta,pack_stat, andsub_file_stat(all of them are metadata of a DTFile). The system calls and small reads hurt performance.
https://github.com/pingcap/tics/blob/c72c331726b45197f45a8d2af766070b97b2b25f/dbms/src/Storages/DeltaMerge/File/DMFile.cpp#L262-L334 - There are a lots of
isSingleFileMode()/if (mode == Mode::FOLDER)inside DTFile implementation, which make it messy. We can split the logic into two subclassesDTFileDirectory,DTFileSingle, and make the codes more clear.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.