Negative value in the Prometheus metric cache_pinned_usage

Expected Behavior

The cache_pinned_usage prometheus metric reflects the actual size of pinned entries in the history cache.

Actual Behavior

The cache_pinned_usage metric can take a negative value then history.cacheSizeBasedLimit is set to true:

cache_pinned_usage{cache_type="mutablestate",service_name="history",type="history"} -4.7310617e+07

Probable cause

It is possible that the issue is related to the fact that the metric value is increased by the size of the entry only at the moment it is created:

https://github.com/temporalio/temporal/blob/bd7a7c23b0ee0c3ec3d9fa35c83145931af2b4b5/common/cache/lru.go#L441-L444

After that, the same entry may be modified in a way that its size becomes larger.

And when the entry is released, its current size will be subtracted from the cache’s pinnedSize and reflected in the metric.

https://github.com/temporalio/temporal/blob/bd7a7c23b0ee0c3ec3d9fa35c83145931af2b4b5/common/cache/lru.go#L269-L272

It is possible that situations where the entry size at the time of its release is larger than at the time of its creation are exactly what leads to the metric having a negative value.

Steps to Reproduce the Problem

1. Enable option history.cacheSizeBasedLimit
2. After some time, get the value of the cache_pinned_usage metric

Specifications

• Version: v1.29.1
• Platform: Linux/amd64 (RKE2 Kubernetes)