brain-score / brain-score/vision
submit multiple models in one zip file?
- Dominant language
- Python
- Stars
- 193
- Forks
- 105
- Avg merge
- 10h 48m
- Merged PRs (30d)
- 10
Description
It seems that currently the users need to upload a complete directory structure for each model instance they want to get a score.
However, users often want to submit multiple similar models together. For example, `model_0`, `model_1`, and `model_2` have the same structure but different weights (eg. they could be trained with different augmented versions of the same dataset). Writing the entire directory structure for each of them could be repetitive.
Would it be possible for the user to submit multiple models in one zip file (one directory)? I saw that on these tutorials [1](https://www.brain-score.org/tutorial/deepdive_1) [2](https://www.brain-score.org/tutorial/deepdive_2) there seem to be a function that do that:
```
def get_model_list():
return ['my_custom_model']
```
However, this function is not used anywhere that I know of.
Enabling multiple model submissions in one file could look something like this:
In the `__init__.py` file
```python
from .model import get_model, LAYERS, get_model_list
for model in get_model_list():
model_registry[model] = lambda: ModelCommitment(
identifier=model,
activations_model=get_model(model),
layers=LAYERS)
```
In the `model.py` file
```python
def get_model_list():
return ['my_custom_model_0', 'my_custom_model_0', 'my_custom_model_0']
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.