ClickHouse / ClickHouse/clickhouse-connect

Serialization of Array types in numpy slow

Open
#1,008 5 comments 1 reaction 1 assignee Claimed by @joe-clickhouse View on GitHub
enhancement
Dominant language
Python
Stars
521
Forks
159
Avg merge
1d 8h
Merged PRs (30d)
43

Description

**Is your feature request related to a problem? Please describe.**
When we started using query_np, we saw a significant difference in querying scalar columns vs. array columns. This can be easily reproduced with the following setup:

```
create database if not exists test;
create table test.scalar
(
u64 UInt64,
scalar UInt64
) ENGINE = MergeTree()
ORDER BY u64;
create table test.array
(
u64 UInt64,
arr Array(UInt64)
) ENGINE = MergeTree()
ORDER BY u64;
insert into test.scalar select number, number from numbers(1000000);
insert into test.array select number, [number] from numbers(1000000);
```

If we now run:
```
a_np = functools.partial(client.query_np, "Select * from test.scalar")
b_np = functools.partial(client.query_np, "Select * from test.array")
```

We see the following timing averages:
```
>>> timeit.timeit(a_np, number=100)/100
0.08031970925999303
>>> timeit.timeit(b_np, number=100)/100
0.31934394390999843
```

Compare this to query_raw:
```
>>> a_raw = functools.partial(client.raw_query, "Select * from test.scalar")
>>> b_raw = functools.partial(client.raw_query, "Select * from test.array")
>>> timeit.timeit(a_raw, number=100)/100
0.0437763816900042
>>> timeit.timeit(b_raw, number=100)/100
0.04174081839999417
```

query_raw performs about the same within errors while query_np is 3-4x more expensive when querying arrays over scalar columns.

The main issue here is that array columns gets serialized into a NDArray[object], with the content being python lists. On the wire it seems like the array column is encoded with two sequences:

- A flat array of values
- A flat array of offsets into the values list

when serializing, this columnar format needs to be converted into the row like format that is encoded by NDArray[list]. This and the relatively high overhead in python list creations defeats any performance gains from serializing into a numpy array. Downstream its also harder to benefit from any numpy features like vectorized processing and other SIMD optimizations.

Also see the two attached graphs that compare query_raw vs query_np performance on both.

Image

Image

**Describe the solution you'd like**
It would be nice to have an interface that exposes the tuple of `NDArray[column_dtype]` (values) and `NDArray[np.uint64]` (offsets) over a single NDArray[list] type alongside the existing interface. Right now, I built a custom native parser to achieve this. It makes the fetch way faster and allows me to use native numpy datatypes downstream. However, it would be nice if I could use query_np :)

**Describe alternatives you've considered**
I tried query_arrow which fetches the arrow format and serialized directly into an arrow table. This obviously does not suffer from the same limitation but introduces a ~150mb dependency for arrow tables which we convert into numpy arrays right after the fetch.

I also looked into ways to represent this as a singular native numpy array but I couldn't find any way to represent a fixed array with variable sized list members.

**Additional context**
I am happy to contribute a PR. However, I wanted to take the opportunity to confirm my findings and discuss possible solutions. Especially on the interface level, I do not have a good idea right now how this could be expressed without loosing typing guarantees or make the interfaces needlessly complex.

Thanks for looking into this. Let me know what you think!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.