cockroachdb / cockroachdb/cockroach
Periodically log tenant client token usage
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
While investigating an issue with a customer PoC, we discovered a situation where a Standard tenant client's view of token usage appears to be much greater than the server's view, resulting in cluster utilization that is approximately 2x provisioned.
We lack the visibility to see precisely what the client's view of the world is, so we'd like to periodically log state. We note that there are other systems that log approximately every 10s:
```
I250718 09:40:10.603281 804 2@util/log/event_log.go:90 ⋮ [T1,Vsystem,n1] 68567 ={"Timestamp":1752831610603275000,"EventType":"runtime_stats","MemRSSBytes":680230912,"GoroutineCount":491,"MemStackSysBytes":6619136,"GoAllocBytes":545863384,"GoTotalBytes":786066504,"HeapFragmentBytes":73992488,"HeapReservedBytes":98410496,"HeapReleasedBytes":680124416,"CGoAllocBytes":145166912,"CGoTotalBytes":185991168,"CGoCallRate":259.8162,"CPUUserPercent":0.110006854,"CPUSysPercent":0.15000935,"GCRunCount":2951,"NetHostRecvBytes":647835,"NetHostSendBytes":413774}
I250718 09:40:10.603695 804 2@server/status/runtime_log.go:43 ⋮ [T1,Vsystem,n1] 68568 runtime stats: 649 MiB RSS, 491 goroutines (stacks: 6.3 MiB), 521 MiB/750 MiB Go alloc/total (heap fragmentation: 71 MiB, heap reserved: 94 MiB, heap released: 649 MiB), 138 MiB/177 MiB CGO alloc/total (259.8 CGO/sec), 0.1/0.2 %(u/s)time, 0.0 %gc (2951x), 633 KiB/404 KiB (r/w)net
```
It might be nice to coalesce this periodic logging of state into a single facility controlled by a cluster setting.
Jira issue: CRDB-52748
Contributor guide
Assessment
This issue has not been assessed yet.