cockroachdb / cockroachdb/cockroach

Periodically log tenant client token usage

Open
#150,536 0 comments 0 reactions 0 assignees View on GitHub
A-multitenancy C-enhancement T-sql-queries
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

While investigating an issue with a customer PoC, we discovered a situation where a Standard tenant client's view of token usage appears to be much greater than the server's view, resulting in cluster utilization that is approximately 2x provisioned.

We lack the visibility to see precisely what the client's view of the world is, so we'd like to periodically log state. We note that there are other systems that log approximately every 10s:

```
I250718 09:40:10.603281 804 2@util/log/event_log.go:90 ⋮ [T1,Vsystem,n1] 68567 ={"Timestamp":1752831610603275000,"EventType":"runtime_stats","MemRSSBytes":680230912,"GoroutineCount":491,"MemStackSysBytes":6619136,"GoAllocBytes":545863384,"GoTotalBytes":786066504,"HeapFragmentBytes":73992488,"HeapReservedBytes":98410496,"HeapReleasedBytes":680124416,"CGoAllocBytes":145166912,"CGoTotalBytes":185991168,"CGoCallRate":259.8162,"CPUUserPercent":0.110006854,"CPUSysPercent":0.15000935,"GCRunCount":2951,"NetHostRecvBytes":647835,"NetHostSendBytes":413774}
I250718 09:40:10.603695 804 2@server/status/runtime_log.go:43 ⋮ [T1,Vsystem,n1] 68568 runtime stats: 649 MiB RSS, 491 goroutines (stacks: 6.3 MiB), 521 MiB/750 MiB Go alloc/total (heap fragmentation: 71 MiB, heap reserved: 94 MiB, heap released: 649 MiB), 138 MiB/177 MiB CGO alloc/total (259.8 CGO/sec), 0.1/0.2 %(u/s)time, 0.0 %gc (2951x), 633 KiB/404 KiB (r/w)net
```

It might be nice to coalesce this periodic logging of state into a single facility controlled by a cluster setting.

Jira issue: CRDB-52748

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.