brain-score / brain-score/brain-score.web

Add Support for Data Plugins in Large File Upload Portal

Open
#487 0 comments 0 reactions 1 assignee Claimed by @mike-ferguson View on GitHub
Dominant language
JavaScript
Stars
3
Forks
7
Avg merge
7m
Merged PRs (30d)
1

Description

Currently, large file uploads support model weight uploads. At first glance, it seems straight forward to update to enable Data upload. However, after testing this, I am running into the following issues:

### 1. StimulusSet download requires the ZIP and CSV to be downloaded at the same time and loaded to the correct location.

Currently, you can select the data folder and upload the file to the data folder under the user subdirectory. However, right now, it requires the user to submit the ZIP and the CSV independently. It is not entirely straight forward how the provided Clipboard Code is then used with the standard `stimulus_set_registry` as the current process is loading both the ZIP and CSV via `load_stimulus_set_from_s3()`.

### 2. load_file() is inadequate for dataAssembly.

I think this one is self-explanatory as the `load_assembly_from_s3()` performs additional functionality like attaching the stimulus set if loader is passed.

## Current Workarounds (and likely solution)

I've managed to still use the Data folder via LFU by using the following information to construct the standard `load_{stimulus_set/assembly}_from_s3()`.

#### load_assembly_from_s3()
- identifier <--- from `package_data_assembly_locally()`
- version_id <--- from LFU provided version_id
- sha1 <--- from `package_data_assembly_locally()`
- bucket <--- from LFU by appending `bucket` + `folder_name`
- cls <--- from `package_data_assembly_locally()`

#### load_stimulus_set_from_s3()
- identifier <--- from `package_stimulus_set_locally()`
- bucket <--- from LFU of either zip/csv by appending `bucket` + `folder_name`
- csv_sha1 <--- from `package_stimulus_set_locally()`
- zip_sha1 <--- from `package_stimulus_set_locally()`
- csv_version_id <--- from LFU of csv provided version_id
- zip_version_id <--- from LFU of zip provided version_id
- filename_prefix = "stimulus_"

## Recommendation

When data folder is selected in LFU, it then shows option for StimulusSet OR Assembly. If StimulusSet is selected, user must upload both CSV and ZIP at the same time. If Assembly is selected, user must upload the NC file.

In the central profile page, the User Uploaded Files can continue to show all files individually. The dataAssembly (NC file), will provide the exact code {excluding `sha1`} for the user to run `load_assembly_from_s3()`. Both the CSV and ZIP file's code copy button will provide the same code. That is the exact code {excluding `csv_sha1` and `zip_sha1`} for the user to run `load_stimulus_set_from_s3()`.

We also include a note under the User Uploaded Files table indicating that if a sha1 that was produced during local packaging does not work with the provided code, then there was a `Data Integrity Issue` and worth exploring or retrying.

A final note is that we should remove the `/benchmark/` folder as there isn't a meaningful enough difference between that and the data folder.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.