brain-score / brain-score/vision

Handle multi-plugin submissions elegantly

Open
#734 0 comments 0 reactions 1 assignee Claimed by @deirdre-k View on GitHub
Dominant language
Python
Stars
193
Forks
105
Avg merge
10h 48m
Merged PRs (30d)
10

Description

Right now we cap web submissions at 5 plugins to prevent tests from timing out (see [vision issue #91](https://github.com/brain-score/core/issues/91)). This is a burden on users, and additionally will still occasionally time out.

Long term, we would like to handle larger submissions with some considerations for what sorts of bulk submissions we allow.

* [ ] Consider batching large submissions into either
* multiple PRs
* separate jobs on Travis

* [ ] Consider whether we want to add checking to only allow certain kinds of bulk submission

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.