brain-score / brain-score/vision
Kar2019 testing score assertion errors
- Dominant language
- Python
- Stars
- 193
- Forks
- 105
- Avg merge
- 10h 48m
- Merged PRs (30d)
- 10
Description
Score assertion errors are present in the [kar2019 benchmark testing](https://github.com/brain-score/vision/blob/master/brainscore_vision/benchmarks/kar2019/test.py). See below for an example:
```
def test_Kar2019ost_cornet_s():
benchmark = load_benchmark('Kar2019-ost')
filename = 'cornet_s-kar2019.nc'
filepath = Path(__file__).parent / filename
s3.download_file_if_not_exists(local_path=filepath,
bucket='brainscore-unittests', remote_filepath=f'tests/test_benchmarks/{filename}')
precomputed_features = NeuroidAssembly.from_files(
filepath,
stimulus_set_identifier=benchmark._assembly.stimulus_set.identifier,
stimulus_set=benchmark._assembly.stimulus_set)
precomputed_features = PrecomputedFeatures(precomputed_features, visual_degrees=8)
# score
score = benchmark(precomputed_features).raw
> assert score == approx(.316, abs=.005)
E assert \narray(0.2408883)\nAttributes:\n raw: \narray([0.14566184, 0.12413281, 0.23...\n error: \narray(0.02970973)
E Expected: 0.316 ± 5.0e-03
test.py:35: AssertionError
```
This assertion error is with brainscore version 2.0.0, whereas when using python 3.11 and the updated brainscore version, the output score is `0.23875906` which still causes an assertion error. This is possibly due to the continued use of tensorflow within the [ost metric](https://github.com/brain-score/vision/blob/master/brainscore_vision/metrics/ost/metric.py).
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.