cockroachdb / cockroachdb/cockroach

sql/logging: ability to throttle high-output logging events

Open
#137,037 2 comments 0 reactions 0 assignees View on GitHub
A-logging C-enhancement T-sql-queries
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

A customer had an issue where a team deployed a query that was malformed and/or couldn't be parsed by CRDB.

This resulted in possibly billions of log entries being generated which caused the downstream logging gateway to get overwhelmed.

Due to the structure of the query, each failed execution generated no less than 242 log entries. Looking at a single node, the customer had ~104k failed queries. Extrapolating that to 99 nodes, that would be ~10.2M failed queries. And at 242 rows each, that resulted in ~2.5B log entries over a short period of time.

We should investigate a way to throttle these types of events so as not to affect the node disks or downstream gateways.

Suggestion from the customer: Coalesce each failed query to a single log entry rather than 242 entries.

Jira issue: CRDB-45368

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.