cockroachdb / cockroachdb/cockroach

logging: implement character limit on emitted logs

Open
#126,236 1 comment 0 reactions 0 assignees View on GitHub
C-bug
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

CockroachDB does not have a character limit on emitted log lines which prevents logging sinks like Fluent Bit from receiving the entire log line as a single event.

Here's one such long log event, redacted where the majority of the characters occur.

```json
{
"tag": "cockroach.telemetry",
"channel_numeric": 12,
"channel": "TELEMETRY",
"timestamp": "1719264767.684544406",
"cluster_id": "0aca882a-4a60-4b86-8b19-dae7a612d826",
"tenant_id": 10,
"tenant_name": "cluster-10",
"instance_id": 1,
"version": "v24.1.1",
"severity_numeric": 1,
"severity": "INFO",
"goroutine": 261006378,
"file": "util/log/event_log.go",
"line": 32,
"entry_counter": 4517475,
"redactable": 1,
"tags": {
"n": "sql1",
"peer": "‹10.0.5.245:51142›",
"client": "34.173.198.153:35120",
"hostssl": "",
"user": "‹qualification-workload›"
},
"event": {
"Timestamp": 1719264767676765008,
"EventType": "sampled_query",
"Statement": "UPSERT INTO \"\".\"\".kv(k, v) VALUES ($1, $2), ($3, $4), ($5, $6), ($7, $8), ($9, $10), ($11, $12), ($13, $14), ($15, $16), ($17, $18), ($19, $20)",
"Tag": "INSERT",
"User": "‹qualification-workload›",
"ApplicationName": "kv",
"PlaceholderValues": [
"‹-1869857217101111026›",
"REDACTED LONG STRING",
"‹4777978783825699802›",
"REDACTED LONG STRING",
"‹603098788185010468›",
"REDACTED LONG STRING",
"‹1326509987868430009›",
"REDACTED LONG STRING",
"‹8234914939766267954›",
"REDACTED LONG STRING",
"‹433961672278954175›",
"REDACTED LONG STRING",
"‹462940093285206272›",
"REDACTED LONG STRING",
"‹8779261853781541214›",
"REDACTED LONG STRING",
"‹-2391050638372379906›",
"REDACTED LONG STRING",
"‹248356239080483531›",
"REDACTED LONG STRING",
],
"ExecMode": "exec",
"NumRows": 10,
"Age": 6.832211,
"StmtPosInTxn": 1,
"SkippedQueries": 6,
"CostEstimate": 0.12,
"Distribution": "local",
"PlanGist": "AgIUBAUGIuYBAQ==",
"SessionID": "17dc0e2864b9cad50000000000000001",
"Database": "kv",
"StatementID": "17dc0e32783991f80000000000000001",
"TransactionID": "7d765250-befb-448e-871d-1efb45c19857",
"StatementFingerprintID": 11236207118300501284,
"RowsWritten": 10,
"ServiceLatencyNanos": 6620499,
"OverheadLatencyNanos": 2040,
"RunLatencyNanos": 5937285,
"PlanLatencyNanos": 681174,
"SchemaChangerMode": "none"
}
}
```

This log event was 206584 bytes in size due to the large `event.PlaceholderValues` values.

```sh
> wc -c /tmp/large-log-line.log
206584 /tmp/large-log-line.log
```

**To Reproduce**

This was observed on a staging CockroachDB Cloud cluster (`0aca882a-4a60-4b86-8b19-dae7a612d826`).

**Expected behavior**

CockroachDB should impose a limit on the number of character a given log line can be, preferring to split a single log event across multiple events instead.

**Additional data / screenshots**

N/A

**Environment:**
- CockroachDB version v24.1.1
- Server OS: Red Hat Enterprise Linux 9.4 (Plow), kernel 5.15.146+

**Additional context**

What was the impact?

The Fluent Bit sink CockroachDB was configured to send logs to was unable to receive logs over TCP, preventing logs from being sent to a centralized system.

```
[2024/06/26 01:46:11] [ warn] [input:tcp:tcp-cockroach-unredacted] invalid JSON message, skipping
[2024/06/26 01:46:12] [ warn] [input:tcp:tcp-cockroach-unredacted] fd=63 incoming data exceeds 'Buffer_Size' (32 KB)
```

Jira issue: CRDB-39806

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.