brain-score / brain-score/result_caching
Toggling RESULTCACHING_DISABLE changes ridge scores (~1.5%)
- Dominant language
- Python
- Stars
- 3
- Forks
- 13
- PR merge metrics
- No merged PRs in 30d
Description
## What
Turning result_caching off vs on is **not score-neutral** for cross-validated ridge benchmarks.
`alexnet` x `Zerbe2026_fmri.IT-tau-ridgecv`, identical code and container, differing only in caching:
```
RESULTCACHING_DISABLE=1 (all caching off) 0.27247561041393065
caching enabled 0.2765740756894441
```
~1.5% — far too large for float noise, and reproducible (two independent runs of the enabled configuration agreed to the last digit).
Two RDM benchmarks under the same conditions were **unaffected**:
```
Zerbe2026_fmri.V1-rdm-pearson 0.3004423391786891 identical both ways
Zerbe2026_fmri.IT-rdm-pearson 0.3745153062392548 identical both ways
```
## It is not the activation cache
Ruled out directly. With only the activation store disabled —
```
RESULTCACHING_DISABLE=brainscore_vision.model_helpers.activations
```
— all three of {activations off, cached cold, cached warm} give **0.2765740756894441**, and a fingerprinting run found the activation assemblies **bit-identical** between fresh and cached (same `values_sha`, dtype, shape, across 10 `look_at` calls). The netCDF/S3 round-trip is faithful.
So the difference comes from one of the *other* `@store`-decorated functions that `=1` also disables. The prime suspect is `brainscore_vision.benchmark_helpers.screen._place_on_screen`, which caches the stimulus-degree conversion: if the cached `StimulusSet` differs from the recomputed one in row order, ridge's CV folds change — a genuinely different fit — while RDM, which aligns on `stimulus_id`, would not notice. The rdm-vs-ridge asymmetry is consistent with that.
## Why it matters
1. **A/B comparisons across cache settings are invalid** for ridge-family benchmarks. Anyone validating a caching change by toggling `=1` will see drift that has nothing to do with what they changed. This cost us a wrong conclusion — we initially reported the activation cache as unsafe.
2. Production Batch sets `RESULTCACHING_DISABLE=1` unless the activation cache overrides it (`brainscore_scoring/batch.py::_build_env_vars`), so **enabling the cache flips this setting** and may shift ridge scores for reasons unrelated to activations.
## Suggested next step
Determine whether `_place_on_screen`'s cached return differs from its recomputed one in stimulus order (or anything else). If it does, either make the ordering deterministic across the cache boundary, or have benchmarks sort explicitly before splitting so CV folds cannot depend on it.
Not urgent for correctness of *cached* runs — those are self-consistent — but it is a trap for anyone measuring cache effects, and it means the two configurations are not interchangeable.
## Reproducing
Three passes in separate containers, one benchmark, varying only the disable prefix. Scripts and raw output:
`s3://brainscore-storage/scratch/phase4-2026-08-07/` (the flawed `=1` control) and
`s3://brainscore-storage/scratch/phase4b-2026-08-07/` (the corrected one).
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with brainscore_vision.benchmark_helpers.screen._place_on_screen and compare its cached and recomputed StimulusSet returns, especially row order; trace how RESULTCACHING_DISABLE is built in brainscore_scoring/batch.py::_build_env_vars. Use the supplied S3 reproductions to verify the ridge score drift, and consider the work done when the cache boundary preserves ordering or benchmark folds no longer depend on it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100