cockroachdb / cockroachdb/cockroach
sql/logging: ability to throttle high-output logging events
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
A customer had an issue where a team deployed a query that was malformed and/or couldn't be parsed by CRDB.
This resulted in possibly billions of log entries being generated which caused the downstream logging gateway to get overwhelmed.
Due to the structure of the query, each failed execution generated no less than 242 log entries. Looking at a single node, the customer had ~104k failed queries. Extrapolating that to 99 nodes, that would be ~10.2M failed queries. And at 242 rows each, that resulted in ~2.5B log entries over a short period of time.
We should investigate a way to throttle these types of events so as not to affect the node disks or downstream gateways.
Suggestion from the customer: Coalesce each failed query to a single log entry rather than 242 entries.
Jira issue: CRDB-45368
Contributor guide
Assessment
This issue has not been assessed yet.