google-deepmind / google-deepmind/gemma

Add cache usage property to ChatSampler and Sampler

Open
#471 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
5.7k
Forks
1k
Avg merge
10h 33m
Merged PRs (30d)
2

Description

## Cache Usage Monitoring Enhancement

### Overview
Previously, a TODO was marked to add a property showing how much of the cache is used. This is useful for debugging and monitoring cache usage during multi-turn conversations.

---

### Previous State
- No easy way to check cache usage without accessing internal `last_state` directly
- Users had to manually compute `state.used_cache_length / cache_length`

---

### Changes

#### Added Cache Usage Monitoring
Cache usage monitoring was added to both sampler classes:

- **`ChatSampler.cache_usage` property**
- Returns a `(used, total)` tuple
- Returns `None` if no sampling has been performed yet

- **`Sampler.get_cache_usage()` static method — For stateless usage with the low-level API**
- Takes a SamplingState and cache_length as parameters

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.