brain-score / brain-score/vision

result_caching/copying looses StimulusSet attributes that are not in `pd.DataFrame._metadata`

Open
#2,282 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
193
Forks
105
Avg merge
10h 48m
Merged PRs (30d)
10

Description

The `_place_on_screen` function in [screen.py](https://github.com/brain-score/vision/blob/master/brainscore_vision/benchmark_helpers/screen.py) adds the paths to the untransformed images after placing images on the screen as a custom `StimulusSet` attribute:
[l.72](https://github.com/brain-score/vision/blob/26133e4fb54810c42afd8c2dd2be91abc2241398/brainscore_vision/benchmark_helpers/screen.py#L72C1-L73C1) `converted_stimuli.original_paths = copy.deepcopy(stimulus_set.stimulus_paths)`
It seems to me that when `result_caching`'s `@store` pickles and unpickles the `StimulusSet`, all attributes not defined in `_metadata` [here in brainscore_core](https://github.com/brain-score/core/blob/main/brainscore_core/supported_data_standards/brainio/stimuli.py) are lost.
Thus, the transformed stimulus set is different when it is computed vs when it is loaded from cache.

This is also the case when the StimulusSet is copied or sliced, [see this older StackOverflow entry](https://stackoverflow.com/questions/50372509/why-are-attributes-lost-after-copying-a-pandas-dataframe).

I believe this behavior is completely silent right now, because a) nothing actually uses the original paths, b) there is no test that checks whether `result_caching` returns identical attributes. Therefore, there is really no urgency to address this right now, but I'd still say it is unexpected/unintended behavior.

Steps to reproduce: load a transformed stimulus set from `.result_caching/brainscore_vision.benchmark_helpers.screen._place_on_screen` and check for `original_paths`

Possible ways to address this might be to either
- extend `_metadata`
- use `converted_stimuli.attrs['original_paths']` (it seems to me that then the attribute will survive pickling, but it is not a proper attribute so any `hasattr` check will still fail (?))
- or address it over at result_caching by making a custom store decorator for DataFrames/StimulusSets that preserves any attribute

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with _place_on_screen in brainscore_vision/benchmark_helpers/screen.py and the StimulusSet definition in brainscore_core's stimuli.py. Reproduce the issue by loading the transformed stimulus set from result_caching and checking original_paths, then compare copied or sliced sets. Done means the attribute is preserved consistently after caching, copying, or slicing, with a regression test covering the behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.