feldera / feldera/feldera

Kafka transport configuration defers too many errors

Open
#2,149 1 comment 1 reaction 0 assignees View on GitHub
connectors
Dominant language
Rust
Stars
2.1k
Forks
153
Avg merge
2d 11h
Merged PRs (30d)
129

Description

The design for the Kafka input and output transport configurations have a flattened map, like this:

```
/// Options passed directly to `rdkafka`.
///
/// See [`librdkafka` options](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md)
/// used to configure the Kafka producer.
#[serde(flatten)]
pub kafka_options: BTreeMap,
```

This means that if the user adds some configuration key-value pair that neither rdkafka nor Feldera understands, it will get passed to rdkafka at runtime and then fail. It would be better for this to happen only if the user intended it for rdkafka, so that options meant for Feldera itself but misspelled could be detected at configuration time instead of at pipeline startup time.

This could be done by just removing `#[serde(flatten)]`, at the cost of incompatibly changing the configuration schema.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.