ClickHouse / ClickHouse/ClickHouse

Clickhouse unconditionally loads all data from Arrow files in S3

Open
#82,429 0 comments 0 reactions 0 assignees View on GitHub
comp-formats external performance
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

### Company or project name

MGT

### Describe the unexpected behaviour

When I load data from arrow files that are in s3, I expect that only the projected columns be sent over the network. However, clickhouse loads the entire arrow file (including columns that are unused). For example, I am using a query that looks like this (I'm removed a lot of columns to simplify):

```
insert into firewall_traffic
select timestamp,destination_ip,source_ip
from s3('s3://bucketname/firewall_logs.arrow', 'aws_access_key_id', 'aws_secret_access_key', 'Arrow');
```

The arrow file has another column in addition to these three: the original log. In the applications that I'm working on, it is critical that the original logs be preserved somewhere, but they aren't usually needed. So when the data is loaded into clickhouse, these can be omitted.

Since Clickhouse reads the entire arrow file from s3 before processing it, the AWS egress byte count that is several times higher than it needs to be. (S3 supports byte-range queries and has for a long time)

The reason I believe that Clickhouse loads the entire file is because, in the query log, I see that `read_bytes` is about 55MB regardless of whether I perform `SELECT timestamp, destination_ip, source_ip` or `SELECT timestamp, destination_ip, source_ip, original_log`. It should be about an order of magnitude lower if the original log is skipped.

I can understand why it might be difficult to support this, and if it's not an optimization that Clickhouse intends to support, it would be nice to explicitly call this out the documentation about Arrow (and presumably Parquet since I suspect the same issue is present there).

### Which ClickHouse versions are affected?

all versions

### How to reproduce

Load data from S3 into a clickhouse table, then check the query log for `read_bytes`.

### Expected behavior

_No response_

### Error message and/or stacktrace

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.