pingcap / pingcap/tiflash

Redesign the single file format for DTFile

Open
#1,798 2 comments 0 reactions 1 assignee View on GitHub

@lidezhu is already working on this.

Since Apr 23, 2021.

Dominant language
C++
Stars
1k
Forks
423
Avg merge
1d 15h
Merged PRs (30d)
24

Description

We propose a "single file mode" for DTFile to store the snapshot generated / snapshot files for ingesting. Storing data in a single file is easy for transferring data between nodes.
Single File Format

But there are some problems for current design and implementation:

  1. The file format is not efficient. For example, we need to open and read the same file four times to get Footer, meta, pack_stat, and sub_file_stat (all of them are metadata of a DTFile). The system calls and small reads hurt performance.
    https://github.com/pingcap/tics/blob/c72c331726b45197f45a8d2af766070b97b2b25f/dbms/src/Storages/DeltaMerge/File/DMFile.cpp#L262-L334
  2. There are a lots of isSingleFileMode() / if (mode == Mode::FOLDER) inside DTFile implementation, which make it messy. We can split the logic into two subclasses DTFileDirectory, DTFileSingle, and make the codes more clear.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.