ContextLab / ContextLab/hypertools

grand challenge: streaming brain decoding

Open
#113 4 comments 0 reactions 0 assignees View on GitHub
awesome enhancement help wanted high effort pie in the sky
Dominant language
Python
Stars
1.9k
Forks
164
Avg merge
23h 31m
Merged PRs (30d)
1

Description

Achieving this grand challenge requires:
1. Support for [streaming data](https://github.com/ContextLab/hypertools/issues/101)
2. [Interactive feature/event labels](https://github.com/ContextLab/hypertools/issues/111)
3. [On-the-fly decoding](https://github.com/ContextLab/hypertools/issues/112)
4. Reading in brain data on the fly, e.g. from an OpenBCI device (e.g. see [this](https://github.com/OpenBCI/OpenBCI_Python) project)

Here's the vision:
The user wears their brain recording device, streaming data into hypertools. Periodically, they focus hard on imagining a word (e.g. picture an apple as intensely as possible for a few seconds). As this happens, they [press spacebar and tag that brain pattern/event with the label "apple"](https://github.com/ContextLab/hypertools/issues/111). This repeats for dozens (hundreds?) of words, and several presentations of each word. The decoding model (labeled brain patterns) is saved out to disk.

Now we switch to "decode" mode. Load in the decoding model and start streaming data from the headset again. Now the user picks a word from the labeled set and focuses hard on bringing that word to mind. Imagine their shock and delight when their brain trajectory moves to the appropriate labeled point, and [the word they were thinking of is highlighted on the display](https://github.com/ContextLab/hypertools/issues/112)!

Here's another (related) vision:
This tool could be used as a benchmark for brain decoding challenges. For example, suppose someone writes a `feature_extractor` function for translating raw brain data into arbitrary features (power spectra, some sort of deep neural network's re-representation of the data, classifier outputs, etc.). In other words, the `feature_extractor` allows us to focus in on the features/components of brain activity we think are important.

We also need a decoding function (`decoder`). This could be based on the Euclidean distance between the current brain patterns and labeled patterns, correlation between patterns, etc. The decoder tells us how to map between extracted features and labeled points, ideally in a robust way.

Now brain decoding is a matter of finding the right `feature_extractor` function and `decoder` function.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.