ClickHouse / ClickHouse/ClickStack-helm-charts
Kafka Engine throwing Consumer error: Local: Required feature not supported by broker
- Dominant language
- Shell
- Stars
- 75
- Forks
- 54
- Avg merge
- 3d 19h
- Merged PRs (30d)
- 7
Description
Kafka deployment:
Bitnami kafka helm chart version kafka-32.2.12, Kafka version 4.0.0
Clickhouse deployment:
hyperdx helm chart version hdx-oss-v2-0.6.4, ClickHouse version 24.10.2.80, which to the best of my knowledge runs lirbkafka 2.8.0
Problem:
When attempting to use kafka engine to push messages into clickhouse, No messages are sent and I get the following:
```
│ StorageKafka (syslog_test_table): sasl.kerberos.kinit.cmd configuration parameter is ignored. │
│ StorageKafka (syslog_test_table): Consumer error: Local: Required feature not supported by broker │
│ StorageKafka (syslog_test_table): Consumer error: Local: Required feature not supported by broker │
│ StorageKafka (syslog_test_table): Consumer error: Local: Required feature not supported by broker │
│ StorageKafka (syslog_test_table): Consumer error: Local: Required feature not supported by broker │
│ StorageKafka (syslog_test_table): There were 4 messages with an error │
│ StorageKafka (syslog_test_table): Only errors left
```
the values-override.yaml:
#note, I added the clickhouse resource definitions myself
clickhouse:
resources:
limits:
cpu: 8
memory: 64Gi
requests:
cpu: 6
memory: 32Gi
hyperdx:
appUrl: "https://url-to-hyperdx"
ingress:
enabled: true
ingressClassName: nginx
host: "url-to-hyperdx"
My named_collections I placed directly into config.xml:
```
test-kafka-svc.[my_namespace].svc.cluster.local:9092
clickhouse-topic
cluster_1_clickhouse_consumer
JSONEachRow
PLAINTEXT
all
```
the admin account I used to make the table:
```
admin
default
default
127.0.0.1
::1
10.0.0.0/8
.*\.svc\.cluster\.local$
1
1
1
1
```
The table definition used:
```
CREATE DATABASE IF NOT EXISTS syslogs;
CREATE OR REPLACE TABLE syslogs.syslog_kafka_table
(
LEVEL String,
SOURCEIP String,
ISODATE DateTime,
HOST String,
FACILITY String,
PID String,
MSGID String,
MSG String,
SDATA String,
RAWMSG String,
PROGRAM String
)
ENGINE = Kafka(
'test-kafka-svc.blacksmith.svc.cluster.local:9092',
'clickhouse-topic',
'unique-consumer-group',
'JSONEachRow'
);
```
And here's a message consumed by kcat (that was running lirbkafka 1.7.0) to show that the keys match the schema:
```
{"SOURCEIP":"data",
"SDATA":"{}",
"RAWMSG":"data",
"PROGRAM":"data",
"PID":"data",
"MSGID":"data",
"MSG":"data",
"LEVEL":"notice",
"ISODATE":"data",
"HOST":"data",
"FACILITY":"data"}
```
the error message comes up whenever a materialized view is made to ingest from the table, or whenever someone selects from the table
(i.e SELECT * FROM syslogs.syslog_test_table LIMIT 1;)
I haven't found this error anywhere else. I have consumed messages successfully using a python pod running lirbkafka 2.8.0 (the same one clickhouse uses), I successfully pinged the kafka brokers from the clickhouse pod (using the shell interface), I used another kafka deployment that used another version (the doughgle/kafka-kraft image) and clickhouse throws the same error.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.