ContextLab / ContextLab/hypertools
grand challenge: streaming brain decoding
- Dominant language
- Python
- Stars
- 1.9k
- Forks
- 164
- Avg merge
- 23h 31m
- Merged PRs (30d)
- 1
Description
Achieving this grand challenge requires:
1. Support for [streaming data](https://github.com/ContextLab/hypertools/issues/101)
2. [Interactive feature/event labels](https://github.com/ContextLab/hypertools/issues/111)
3. [On-the-fly decoding](https://github.com/ContextLab/hypertools/issues/112)
4. Reading in brain data on the fly, e.g. from an OpenBCI device (e.g. see [this](https://github.com/OpenBCI/OpenBCI_Python) project)
Here's the vision:
The user wears their brain recording device, streaming data into hypertools. Periodically, they focus hard on imagining a word (e.g. picture an apple as intensely as possible for a few seconds). As this happens, they [press spacebar and tag that brain pattern/event with the label "apple"](https://github.com/ContextLab/hypertools/issues/111). This repeats for dozens (hundreds?) of words, and several presentations of each word. The decoding model (labeled brain patterns) is saved out to disk.
Now we switch to "decode" mode. Load in the decoding model and start streaming data from the headset again. Now the user picks a word from the labeled set and focuses hard on bringing that word to mind. Imagine their shock and delight when their brain trajectory moves to the appropriate labeled point, and [the word they were thinking of is highlighted on the display](https://github.com/ContextLab/hypertools/issues/112)!
Here's another (related) vision:
This tool could be used as a benchmark for brain decoding challenges. For example, suppose someone writes a `feature_extractor` function for translating raw brain data into arbitrary features (power spectra, some sort of deep neural network's re-representation of the data, classifier outputs, etc.). In other words, the `feature_extractor` allows us to focus in on the features/components of brain activity we think are important.
We also need a decoding function (`decoder`). This could be based on the Euclidean distance between the current brain patterns and labeled patterns, correlation between patterns, etc. The decoder tells us how to map between extracted features and labeled points, ideally in a robust way.
Now brain decoding is a matter of finding the right `feature_extractor` function and `decoder` function.
Contributor guide
Assessment
This issue has not been assessed yet.