pingcap / pingcap/tiflash

IColumn::reserve does not work well for string column

Open
#7,495 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

type/enhancement
Dominant language
C++
Stars
1k
Forks
423
Avg merge
1d 15h
Merged PRs (30d)
24

Description

Enhancement

https://github.com/pingcap/tiflash/blob/a1cbb632cc3083d31c13a95231dbf573c2b43ea2/dbms/src/Columns/IColumn.h#L323-L325

IColumn::reserve is used to reserves memory for specified amount of elements. However, for string column, since the element's size is unknown, here is actually no good way to reverve the memory perfectly. Currently implementation of ColumnString::reserve assumes that the element size is ColumnString::APPROX_STRING_SIZE, and APPROX_STRING_SIZE is hard coded as 64. It does not work well because 64 is just a magic number, it has nothing to do with the real size.

There could be two improvements that can be applied to ColumnString::reserve

  • Sometimes, before reserve, we already know information about the element size. An example is function vstackBlocks, so it is possible that we pass a hint to ColumnString::reserve
  • Some string types in TiFlash is actually has a max length when defined in TiDB(for example: char(10), varchar(10), the max length is 10 character), so TiFlash can estimate the element's size based on these information.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with IColumn::reserve in dbms/src/Columns/IColumn.h and inspect ColumnString::reserve, including its use of APPROX_STRING_SIZE. Then trace vstackBlocks and the handling of fixed-length string types such as char(10) and varchar(10); done means reserve can use available element-size information instead of relying only on the hard-coded estimate.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.