ClickHouse / ClickHouse/ClickHouse
Clickhouse unconditionally loads all data from Arrow files in S3
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
### Company or project name
MGT
### Describe the unexpected behaviour
When I load data from arrow files that are in s3, I expect that only the projected columns be sent over the network. However, clickhouse loads the entire arrow file (including columns that are unused). For example, I am using a query that looks like this (I'm removed a lot of columns to simplify):
```
insert into firewall_traffic
select timestamp,destination_ip,source_ip
from s3('s3://bucketname/firewall_logs.arrow', 'aws_access_key_id', 'aws_secret_access_key', 'Arrow');
```
The arrow file has another column in addition to these three: the original log. In the applications that I'm working on, it is critical that the original logs be preserved somewhere, but they aren't usually needed. So when the data is loaded into clickhouse, these can be omitted.
Since Clickhouse reads the entire arrow file from s3 before processing it, the AWS egress byte count that is several times higher than it needs to be. (S3 supports byte-range queries and has for a long time)
The reason I believe that Clickhouse loads the entire file is because, in the query log, I see that `read_bytes` is about 55MB regardless of whether I perform `SELECT timestamp, destination_ip, source_ip` or `SELECT timestamp, destination_ip, source_ip, original_log`. It should be about an order of magnitude lower if the original log is skipped.
I can understand why it might be difficult to support this, and if it's not an optimization that Clickhouse intends to support, it would be nice to explicitly call this out the documentation about Arrow (and presumably Parquet since I suspect the same issue is present there).
### Which ClickHouse versions are affected?
all versions
### How to reproduce
Load data from S3 into a clickhouse table, then check the query log for `read_bytes`.
### Expected behavior
_No response_
### Error message and/or stacktrace
_No response_
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.