cockroachdb / cockroachdb/cockroach

closedts: understand spurious "closed timestamps side-transport connection dropped" errors

Open
#154,346 0 comments 0 reactions 0 assignees View on GitHub
A-kv-rangefeed C-investigation O-postmortem O-support P-3 T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Across our fleet, we see[^1] a significant volume of these logs:

> closed timestamps side-transport connection dropped from node: 9 (grpc: ‹×› [code 1/Canceled])

These are likely spurious, and not caused by network issues.

We saw these errors in an escalation as well, during periods of time in which no nodes were restarting (which might otherwise be a likely source of those errors).

[^1]: https://us5.datadoghq.com/logs?query=service%3Acockroachdb%20%40channel%3A%28DEV%29%20%22closed%20timestamps%20side-transport%20connection%20dropped%22&agg_m=count&agg_m_source=base&agg_t=count&cols=host%2Cservice&fromUser=true&messageDisplay=inline&refresh_mode=sliding&storage=flex_tier&stream_sort=desc&viz=stream&from_ts=1759133960972&to_ts=1759134260972&live=true

Jira issue: CRDB-54887

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.