cockroachdb / cockroachdb/cockroach

obs: fatal logs from roachprod clusters do not appear in datadog

Open
#165,691 0 comments 0 reactions 0 assignees View on GitHub
branch-master C-bug O-testcluster T-supportability
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

When a CockroachDB node on a roachprod cluster crashes due to a fatal error, the fatal log entry does not appear in Datadog. Non-fatal logs (INFO, WARNING, ERROR) from the same node leading up to the crash are present in Datadog as expected. The fatal log is written to the local log file on the node and are not going to datadog.

**Reproduction steps:**
https://docs.google.com/document/d/1KzWw8CUCiCn-n4ArQ2TwS0UtNEKRM07DT_ZaYIFX614/edit?tab=t.0

**Relevant files:**
The logging pipeline from CockroachDB → fluent-bit → Datadog is configured by these files:
- `pkg/roachprod/install/files/cockroachdb-logging.yaml` - CockroachDB logging config. Defines the fluent-bit sink (fluent-servers.fluent-bit) that sends logs over TCP to 127.0.0.1:5170.
- `pkg/roachprod/agents/fluentbit/files/fluent-bit.yaml.tmpl` - Fluent-bit config template. Receives logs on TCP port 5170 and forwards them to Datadog's HTTP intake.
- `pkg/cmd/drtprod/scripts/setup_datadog_cluster` - Script that bootstraps fluent-bit and OpenTelemetry on cluster.

Jira issue: CRDB-61549

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.