IColumn::reserve does not work well for string column
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 1k
- Forks
- 423
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 24
Description
Enhancement
IColumn::reserve is used to reserves memory for specified amount of elements. However, for string column, since the element's size is unknown, here is actually no good way to reverve the memory perfectly. Currently implementation of ColumnString::reserve assumes that the element size is ColumnString::APPROX_STRING_SIZE, and APPROX_STRING_SIZE is hard coded as 64. It does not work well because 64 is just a magic number, it has nothing to do with the real size.
There could be two improvements that can be applied to ColumnString::reserve
- Sometimes, before reserve, we already know information about the element size. An example is function
vstackBlocks, so it is possible that we pass a hint toColumnString::reserve - Some string types in TiFlash is actually has a max length when defined in TiDB(for example: char(10), varchar(10), the max length is 10 character), so TiFlash can estimate the element's size based on these information.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with IColumn::reserve in dbms/src/Columns/IColumn.h and inspect ColumnString::reserve, including its use of APPROX_STRING_SIZE. Then trace vstackBlocks and the handling of fixed-length string types such as char(10) and varchar(10); done means reserve can use available element-size information instead of relying only on the hard-coded estimate.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100