ClickHouse / ClickHouse/ClickHouse

Kafka engine does not work with non-Confluent Avro topics

Open
#71,444 0 comments 0 reactions 0 assignees View on GitHub
comp-message-queues external unfinished code
Dominant language
C++
Stars
49.9k
Forks
9k
Avg merge
21h 32m
Merged PRs (30d)
515

Description

**Describe the unexpected behaviour**
Kafka engine does not work with plain avro topics (i.e. topics without confluent 5-byte-header in messages).
Error message: `SQL Error [1001] [07000]: Code: 1001. DB::Exception: avro::Exception: Invalid data file. Magic does not match: : while parsing Kafka message (topic: test, partition: 0, offset: 196)': While executing Kafka. (STD_EXCEPTION) (version 24.8.2.1)`

**How to reproduce**
* version 24.8
* `CREATE TABLE` statement:
```
CREATE TABLE test.TEST_PLAIN_AVRO (
`intid` Int32,
`longid` Int64)
ENGINE = Kafka
SETTINGS kafka_broker_list = 'kafka:9093',
kafka_topic_list = 'test',
kafka_group_name = 'test.group',
kafka_schema = 'test.avsc',
kafka_format = 'Avro',
kafka_row_delimiter = '\n';
```
Where test.avsc is a file put into /var/lib/clickhouse/format_schemas and containing:
```
{
"type": "record",
"name": "test",
"fields": [
{
"name": "intid",
"type": "int"
},
{
"name": "longid",
"type": "long"
}
]
}
```
**Expected behavior**
Kafka engine successfully read messages

**Error message and/or stacktrace**
`SQL Error [1001] [07000]: Code: 1001. DB::Exception: avro::Exception: Invalid data file. Magic does not match: : while parsing Kafka message (topic: test, partition: 0, offset: 196)': While executing Kafka. (STD_EXCEPTION) (version 24.8.2.1)`

The error shows that ClickHouse is trying to parse the message as a file, because "magic" is the "Obj" header of Avro-format files, but there should not be such headers in Kafka messages.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.