apache / apache/gluten

[CH] Support convert `Char` type of HiveTable to `FixedString` in ClickHouse backend

Open
#6,057 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Description

Now that the char/varchar type of hive table are always convert to string type in clickhouse backend. And the char type in hive table means the field has fixed bytes, which is suitable to convert to `FixedString` type in clickhouse.

And in some suitation, the fixed string type can have better performance that the string type in clickhouse. such as the querys contains `order by`.

Here we defined 2 tables, has the same data, the first one is defined as `test_tbl1(id UInt64, d String)`, and the second is defined as `test_tbl2(id UInt64, d FixedString(20))`, and we do the query `select * from test_tbl order by d limit 100 settings max_threads=1` to compare the performance

In the first table, the result show as below
![image](https://github.com/apache/incubator-gluten/assets/37431499/565b2dc8-aa28-4e0f-abf3-183f2d9e36b0)

In the second table, the result show as below
![image](https://github.com/apache/incubator-gluten/assets/37431499/1c56785f-a96d-449f-9bdd-c2487f31f806)

and we can see that abort 20% ~ 30% performance improve when the table field is defined as `FixedString` type.

Contributor guide

Open the contributing guide

Research direction

No files or tests are named. Start by locating the ClickHouse backend's Hive Char/varchar type-conversion path and review how fixed byte lengths are represented; done means Hive Char fields map to ClickHouse FixedString while varchar and other existing mappings remain unchanged, with coverage for the conversion.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, scala
Domain
databases
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.