ClickHouse / ClickHouse/ClickHouse
Kafka engine does not work with non-Confluent Avro topics
- Dominant language
- C++
- Stars
- 49.9k
- Forks
- 9k
- Avg merge
- 21h 32m
- Merged PRs (30d)
- 515
Description
**Describe the unexpected behaviour**
Kafka engine does not work with plain avro topics (i.e. topics without confluent 5-byte-header in messages).
Error message: `SQL Error [1001] [07000]: Code: 1001. DB::Exception: avro::Exception: Invalid data file. Magic does not match: : while parsing Kafka message (topic: test, partition: 0, offset: 196)': While executing Kafka. (STD_EXCEPTION) (version 24.8.2.1)`
**How to reproduce**
* version 24.8
* `CREATE TABLE` statement:
```
CREATE TABLE test.TEST_PLAIN_AVRO (
`intid` Int32,
`longid` Int64)
ENGINE = Kafka
SETTINGS kafka_broker_list = 'kafka:9093',
kafka_topic_list = 'test',
kafka_group_name = 'test.group',
kafka_schema = 'test.avsc',
kafka_format = 'Avro',
kafka_row_delimiter = '\n';
```
Where test.avsc is a file put into /var/lib/clickhouse/format_schemas and containing:
```
{
"type": "record",
"name": "test",
"fields": [
{
"name": "intid",
"type": "int"
},
{
"name": "longid",
"type": "long"
}
]
}
```
**Expected behavior**
Kafka engine successfully read messages
**Error message and/or stacktrace**
`SQL Error [1001] [07000]: Code: 1001. DB::Exception: avro::Exception: Invalid data file. Magic does not match: : while parsing Kafka message (topic: test, partition: 0, offset: 196)': While executing Kafka. (STD_EXCEPTION) (version 24.8.2.1)`
The error shows that ClickHouse is trying to parse the message as a file, because "magic" is the "Obj" header of Avro-format files, but there should not be such headers in Kafka messages.
Contributor guide
Assessment
This issue has not been assessed yet.