ClickHouse / ClickHouse/ClickStack-helm-charts

Kafka Engine throwing Consumer error: Local: Required feature not supported by broker

Open
#61 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Shell
Stars
75
Forks
54
Avg merge
3d 19h
Merged PRs (30d)
7

Description

Kafka deployment:

Bitnami kafka helm chart version kafka-32.2.12, Kafka version 4.0.0

Clickhouse deployment:

hyperdx helm chart version hdx-oss-v2-0.6.4, ClickHouse version 24.10.2.80, which to the best of my knowledge runs lirbkafka 2.8.0

Problem:

When attempting to use kafka engine to push messages into clickhouse, No messages are sent and I get the following:
```
│ StorageKafka (syslog_test_table): sasl.kerberos.kinit.cmd configuration parameter is ignored. │
│ StorageKafka (syslog_test_table): Consumer error: Local: Required feature not supported by broker │
│ StorageKafka (syslog_test_table): Consumer error: Local: Required feature not supported by broker │
│ StorageKafka (syslog_test_table): Consumer error: Local: Required feature not supported by broker │
│ StorageKafka (syslog_test_table): Consumer error: Local: Required feature not supported by broker │
│ StorageKafka (syslog_test_table): There were 4 messages with an error │
│ StorageKafka (syslog_test_table): Only errors left
```

the values-override.yaml:

#note, I added the clickhouse resource definitions myself
clickhouse:
resources:
limits:
cpu: 8
memory: 64Gi
requests:
cpu: 6
memory: 32Gi
hyperdx:
appUrl: "https://url-to-hyperdx"
ingress:
enabled: true
ingressClassName: nginx
host: "url-to-hyperdx"

My named_collections I placed directly into config.xml:

```



test-kafka-svc.[my_namespace].svc.cluster.local:9092
clickhouse-topic
cluster_1_clickhouse_consumer
JSONEachRow



PLAINTEXT
all

```

the admin account I used to make the table:

```

admin
default
default

127.0.0.1
::1
10.0.0.0/8
.*\.svc\.cluster\.local$


1
1
1
1

```
The table definition used:

```
CREATE DATABASE IF NOT EXISTS syslogs;

CREATE OR REPLACE TABLE syslogs.syslog_kafka_table

(
LEVEL String,
SOURCEIP String,
ISODATE DateTime,
HOST String,
FACILITY String,
PID String,
MSGID String,
MSG String,
SDATA String,
RAWMSG String,
PROGRAM String
)
ENGINE = Kafka(
'test-kafka-svc.blacksmith.svc.cluster.local:9092',
'clickhouse-topic',
'unique-consumer-group',
'JSONEachRow'
);
```

And here's a message consumed by kcat (that was running lirbkafka 1.7.0) to show that the keys match the schema:
```
{"SOURCEIP":"data",
"SDATA":"{}",
"RAWMSG":"data",
"PROGRAM":"data",
"PID":"data",
"MSGID":"data",
"MSG":"data",
"LEVEL":"notice",
"ISODATE":"data",
"HOST":"data",
"FACILITY":"data"}
```

the error message comes up whenever a materialized view is made to ingest from the table, or whenever someone selects from the table
(i.e SELECT * FROM syslogs.syslog_test_table LIMIT 1;)

I haven't found this error anywhere else. I have consumed messages successfully using a python pod running lirbkafka 2.8.0 (the same one clickhouse uses), I successfully pinged the kafka brokers from the clickhouse pod (using the shell interface), I used another kafka deployment that used another version (the doughgle/kafka-kraft image) and clickhouse throws the same error.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.