cockroachdb / cockroachdb/cockroach

Move Kafka messages to its own log

Open
#129,673 2 comments 0 reactions 0 assignees View on GitHub
A-cdc C-enhancement O-support P-3 T-cdc
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Currently any Kafka related messages are logged to the standard dev channel. Kafka consumers constantly loose and regain connections and whenever this happens, log messages are logged. This becomes rather burdensome when troubleshooting issues (especially unrelated to kafka) as the logs are spammed with these messages. This causes the logs to rotate more often than necessary in addition to making the logs larger and more difficult to sift through while troubleshooting unrelated issues to kafka.

It would be ideal if we could log Kafka related messages to its own log channel, much like how pebble and kv have their own logs.

In summary, the ask is:

Please create a separate log for Kafka messages, just like the separate logs for kv and pebble, and remove these messages from the dev channel.

Jira issue: CRDB-41670

Jira issue with more context: CRDB-39717

Epic CRDB-32401

Contributor guide

Open the contributing guide

Research direction

Inspect how the existing pebble and kv logs are configured, then locate the Kafka consumer messages that currently go to the standard dev channel. Confirm the separate Kafka log channel receives those messages while the dev channel no longer does, and verify that log rotation behavior remains appropriate.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kafka
Domain
distributed-systems, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.