cockroachdb / cockroachdb/cockroach
Move Kafka messages to its own log
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
Currently any Kafka related messages are logged to the standard dev channel. Kafka consumers constantly loose and regain connections and whenever this happens, log messages are logged. This becomes rather burdensome when troubleshooting issues (especially unrelated to kafka) as the logs are spammed with these messages. This causes the logs to rotate more often than necessary in addition to making the logs larger and more difficult to sift through while troubleshooting unrelated issues to kafka.
It would be ideal if we could log Kafka related messages to its own log channel, much like how pebble and kv have their own logs.
In summary, the ask is:
Please create a separate log for Kafka messages, just like the separate logs for kv and pebble, and remove these messages from the dev channel.
Jira issue: CRDB-41670
Jira issue with more context: CRDB-39717
Epic CRDB-32401
Contributor guide
Research direction
Inspect how the existing pebble and kv logs are configured, then locate the Kafka consumer messages that currently go to the standard dev channel. Confirm the separate Kafka log channel receives those messages while the dev channel no longer does, and verify that log rotation behavior remains appropriate.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kafka
- Domain
- distributed-systems, observability
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100