brain-score / brain-score/vision

submit multiple models in one zip file?

Open
#646 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
193
Forks
105
Avg merge
10h 48m
Merged PRs (30d)
10

Description

It seems that currently the users need to upload a complete directory structure for each model instance they want to get a score.
submission_structure

However, users often want to submit multiple similar models together. For example, `model_0`, `model_1`, and `model_2` have the same structure but different weights (eg. they could be trained with different augmented versions of the same dataset). Writing the entire directory structure for each of them could be repetitive.

Would it be possible for the user to submit multiple models in one zip file (one directory)? I saw that on these tutorials [1](https://www.brain-score.org/tutorial/deepdive_1) [2](https://www.brain-score.org/tutorial/deepdive_2) there seem to be a function that do that:
```
def get_model_list():
return ['my_custom_model']
```
However, this function is not used anywhere that I know of.

Enabling multiple model submissions in one file could look something like this:

In the `__init__.py` file
```python
from .model import get_model, LAYERS, get_model_list

for model in get_model_list():
model_registry[model] = lambda: ModelCommitment(
identifier=model,
activations_model=get_model(model),
layers=LAYERS)
```

In the `model.py` file
```python
def get_model_list():
return ['my_custom_model_0', 'my_custom_model_0', 'my_custom_model_0']
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.