cockroachdb / cockroachdb/cockroach
obs: fatal logs from roachprod clusters do not appear in datadog
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
When a CockroachDB node on a roachprod cluster crashes due to a fatal error, the fatal log entry does not appear in Datadog. Non-fatal logs (INFO, WARNING, ERROR) from the same node leading up to the crash are present in Datadog as expected. The fatal log is written to the local log file on the node and are not going to datadog.
**Reproduction steps:**
https://docs.google.com/document/d/1KzWw8CUCiCn-n4ArQ2TwS0UtNEKRM07DT_ZaYIFX614/edit?tab=t.0
**Relevant files:**
The logging pipeline from CockroachDB → fluent-bit → Datadog is configured by these files:
- `pkg/roachprod/install/files/cockroachdb-logging.yaml` - CockroachDB logging config. Defines the fluent-bit sink (fluent-servers.fluent-bit) that sends logs over TCP to 127.0.0.1:5170.
- `pkg/roachprod/agents/fluentbit/files/fluent-bit.yaml.tmpl` - Fluent-bit config template. Receives logs on TCP port 5170 and forwards them to Datadog's HTTP intake.
- `pkg/cmd/drtprod/scripts/setup_datadog_cluster` - Script that bootstraps fluent-bit and OpenTelemetry on cluster.
Jira issue: CRDB-61549
Contributor guide
Assessment
This issue has not been assessed yet.