ClickHouse / ClickHouse/clickhouse-cpp

Reading LowCardinality(Nullable(String)) column is not supported (for the data received from ClickHouse)

Open
#504 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
382
Forks
208
Avg merge
2d 19h
Merged PRs (30d)
14

Description

Please see https://github.com/ClickHouse/clickhouse-cpp/pull/339

This one confirms that it works

Perhaps, it would be helpful to know the real type of column at run-time with:


column->GetType().GetName()

Originally posted by @Enmk in #326

The test Map_LowCardinalityTString_LowCardinalityTNullableString might succeed, but this is only true because you create the column with the type ColumnLowCardinalityT<ColumnNullableT<ColumnString>> explicitly in the test.

However, for the real data received from ClickHouse the type of the underlying column will be simply ColumnLowCardinality. This is probably due to the logic implemented in https://github.com/ClickHouse/clickhouse-cpp/pull/182:

case Type::Nullable:
    return std::make_shared<ColumnLowCardinality>(
        std::make_shared<ColumnNullable>(
            CreateColumnFromAst(GetASTChildElement(nested, 0), settings),
            std::make_shared<ColumnUInt8>()
        )
    );

Therefore the dynamic_cast inside column->As<ColumnLowCardinalityT<ColumnNullableT<ColumnString>>>() will yield nullptr.

This is easily illustrated with a simple typeid check (assume that column is an instance of a ColumnRef pointer).

This is what we have for LowCardinality(String) column:

// ColumnRef column;

auto type_name = column->GetType().GetName(); // "LowCardinality(String)"

auto type_id = typeid(*column).name(); // clickhouse::ColumnLowCardinalityT<clickhouse::ColumnString> ("N10clickhouse21ColumnLowCardinalityTINS_12ColumnStringEEE")

And this is the result for LowCardinality(Nullable(String)):

// ColumnRef column;

auto type_name = column->GetType().GetName(); // "LowCardinality(Nullable(String))"

auto type_id = typeid(*column).name(); // clickhouse::ColumnLowCardinality ("N10clickhouse20ColumnLowCardinalityE")

The conclusion is that it's not possible to read LowCardinality(Nullable(String)) column in the current version of clickhouse-cpp (v2.6.1):

  1. column->As<ColumnLowCardinalityT<ColumnNullableT<ColumnString>>>() doesn't work for the reasons explained above.
  2. While column->As<ColumnLowCardinality>() does work, it's of no help: trying to wrap the result into ColumnLowCardinalityT<ColumnNullableT<ColumnString>> will also fail because the nested type is ColumnNullable (not ColumnNullableT<ColumnString>).
  3. Declaring ColumnLowCardinalityT<ColumnNullable> is not possible because there's no ValueType alias in ColumnNullable.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with clickhouse/columns/factory.cpp around the Nullable handling and compare it with ut/roundtrip_tests.cpp at Map_LowCardinalityTString_LowCardinalityTNullableString_. Inspect ColumnRef::As and GetType().GetName() for columns received from ClickHouse, rather than only locally constructed columns. Done means LowCardinality(Nullable(String)) data can be read from a real ClickHouse response and has a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.