confluentinc / confluentinc/kafka-tutorials
Source connector: dynamic topic names pulling information from payload
- Dominant language
- Java
- Stars
- 39
- Forks
- 91
- PR merge metrics
- No merged PRs in 30d
Description
* Chain SMTs: [ExtractTopic](https://docs.confluent.io/platform/current/connect/transforms/extracttopic.html) and then [RegexRouter](https://docs.confluent.io/platform/current/connect/transforms/regexrouter.html)
* Connect-side topic creation: https://cwiki.apache.org/confluence/display/KAFKA/KIP-158%3A+Kafka+Connect+should+all[…]rce+connectors+to+set+topic-specific+settings+for+new+topics
Target KT: extend https://kafka-tutorials.confluent.io/dynamic-output-topic/kstreams.html with this `Kafka` solution?
Considerations:
- Wouldn't that be way too powerful/heavy feature with potentially more harmful side effects than gain? This combination should probably be used with care, but that goes for most features in production-level setups. it's a tough tradeoff between adding guardrails and adding functionality; ultimately, i think this should be fine because...
- What happens if a torrent of messages with unexpected field value (topic name) ? 1000s of new (rogue) topics? That can happen, yes, but if your cluster has acls/rbac/some other authorization mechanism enabled, you can potentially limit the blast radius by locking down topic creation permissions for the connector's principal. something like a suffixed wildcard grant for create permissions might be sufficient for some use cases
- That would imply that with this feature enabled, each message processing would require a check - does the topic already exist, thus adding latency? No, there's caching so that each task only checks for the existence of a topic once over its lifetime
Contributor guide
Assessment
This issue has not been assessed yet.