brain-score / brain-score/vision
StimulusSet data types may clash with DataAssembly data types after s3 upload
- Dominant language
- Python
- Stars
- 193
- Forks
- 105
- Avg merge
- 10h 48m
- Merged PRs (30d)
- 10
Description
When uploading StimulusSets, the `stimulus_id` has to be coded as a string, as otherwise [zip packaging of the StimulusSet fails](https://github.com/brain-score/brainio/blob/main/brainio/packaging.py#L42).
If the `StimulusSet['stimulus_id']` field is a string that contains e.g. only digit characters, when [it is saved as a .csv](https://github.com/brain-score/brainio/blob/main/brainio/packaging.py#L165) and [loaded from s3](https://github.com/brain-score/brainio/blob/main/brainio/stimuli.py#L47), the string datatype for any values that do not contain characters is not respected, resulting in other data types being loaded (as opposed to what were saved).
This is opposed to `DataAssembly`, which [do respect data types](https://github.com/brain-score/brainio/blob/main/brainio/assemblies.py#L461) when being loaded.
When brain-score [merges](https://github.com/brain-score/brainio/blob/main/brainio/assemblies.py#L509) the `StimulusSet` into the `DataAssembly` along the `stimulus_id` dim when [loading the DataAssembly](https://github.com/brain-score/vision/pull/536/files#diff-dd1523e79dd4dcabffc3bc243f8bd2d44191b2a962dd7f947c0db88538522fb1R14),
[interesting errors pop up](http://braintree.mit.edu:8080/job/unittest_plugins/1567/testReport/junit/brainscore_vision.benchmarks.scialom2024.test/TestBehavioral/test_dataset_ceiling_contours_expected_ceiling1_/). This is because while the `stimulus_id` needs to be a string in the `StimulusSet` in order for the `StimulusSet` to be uploaded, the `stimulus_id` also needs to be a csv-inferrable type in the `DataAssembly` (rather than a string) in order for the merging of the two to succeed when loading the `DataAssembly`
This issue is also present for fields that are not `stimulus_id`: `string` types are saved as `.csv` and the data types of values are then inferred on a value-by-value basis. If a column of the `StimulusSet` contains values where some values could be interpreted as strings, and others as integers (e.g., `'condition' = {'100', '35', 'contours', 'RGB'}`), these are inferred differently, resulting in a mix of strings and integers in the `StimulusSet` after loading from s3. This results in [errors](http://braintree.mit.edu:8080/job/unittest_plugins/1574/testReport/junit/brainscore_vision.data.scialom2024.test/TestAssemblies/test_stimulus_set_assembly_alignment_condition_Scialom2024_phosphenes_35_/) on any tests that [test for the integrity of the data](https://github.com/brain-score/vision/pull/536/files#diff-e7188e4d3e58a178bbb10a7bfb930d4fce090da4e805ccbe18c74ce61c368960R69).
Since it does not seem to be possible to fix this like above by enforcing data types on the `DataAssembly` ([since DataArrays don't seem to allow mixed types](https://github.com/pydata/xarray/blob/main/xarray/conventions.py#L100)), the two most reasonable workarounds to this issue seem to be to either code such values explicitly as strings (e.g., `'condition' = {'100a', '35a', 'contours', 'RGB'}` instead of `'condition' = {'100', '35', 'contours', 'RGB'}`), or to [enforce the data types after loading](https://github.com/brain-score/vision/pull/536/files#diff-dd1523e79dd4dcabffc3bc243f8bd2d44191b2a962dd7f947c0db88538522fb1R7).
I would suggest saving the `StimulusSet` in a data format that respects data types, e.g. `xarray netcdf4` instead of `.csv`, or to add more descriptive error messages when aforementioned errors occur.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.