vectordotdev / vectordotdev/vector
Add source/sink metadata logs produced by `rdkafka`
Open
Nobody has claimed this yet.
domain: observability
have: should
sink: kafka
source: kafka
- Dominant language
- Rust
- Stars
- 22.6k
- Forks
- 2.3k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 146
Description
https://github.com/timberio/vector/pull/1918 turns on reporting of rdkafka errors by default. They look like this:
Feb 25 13:28:23.090 ERROR rdkafka::client: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): loc
alhost:9092/1: Disconnected (after 5731ms in state UP)
Feb 25 13:28:23.091 ERROR rdkafka::client: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): Gro
upCoordinator: Disconnected (after 4729ms in state UP)
Feb 25 13:28:28.119 ERROR rdkafka::client: librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): Gro
upCoordinator: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)
Because there might be multiple kafka sources or sinks a single config, these log lines should also contain context like source{name=kafka_1 type=kafka}.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Begin with PR #1918 and the rdkafka error output shown in this issue; trace the Kafka source and sink logging entry points that produce those lines. Done means errors from multiple Kafka sources or sinks include unambiguous source or sink name and type context.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kafka, rust
- Domain
- observability, stream-processing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100