brain-score / brain-score/vision

Kar2019 testing score assertion errors

Open
#1,198 0 comments 0 reactions 2 assignees Claimed by @mike-ferguson View on GitHub
Dominant language
Python
Stars
193
Forks
105
Avg merge
10h 48m
Merged PRs (30d)
10

Description

Score assertion errors are present in the [kar2019 benchmark testing](https://github.com/brain-score/vision/blob/master/brainscore_vision/benchmarks/kar2019/test.py). See below for an example:

```
def test_Kar2019ost_cornet_s():
benchmark = load_benchmark('Kar2019-ost')
filename = 'cornet_s-kar2019.nc'
filepath = Path(__file__).parent / filename
s3.download_file_if_not_exists(local_path=filepath,
bucket='brainscore-unittests', remote_filepath=f'tests/test_benchmarks/{filename}')
precomputed_features = NeuroidAssembly.from_files(
filepath,
stimulus_set_identifier=benchmark._assembly.stimulus_set.identifier,
stimulus_set=benchmark._assembly.stimulus_set)
precomputed_features = PrecomputedFeatures(precomputed_features, visual_degrees=8)
# score
score = benchmark(precomputed_features).raw
> assert score == approx(.316, abs=.005)
E assert \narray(0.2408883)\nAttributes:\n raw: \narray([0.14566184, 0.12413281, 0.23...\n error: \narray(0.02970973)
E Expected: 0.316 ± 5.0e-03

test.py:35: AssertionError
```

This assertion error is with brainscore version 2.0.0, whereas when using python 3.11 and the updated brainscore version, the output score is `0.23875906` which still causes an assertion error. This is possibly due to the continued use of tensorflow within the [ost metric](https://github.com/brain-score/vision/blob/master/brainscore_vision/metrics/ost/metric.py).

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.