BrainJS / BrainJS/brain.js

New Feature: Multithreaded Training

Open
#417 26 comments 28 reactions 3 assignees Claimed by @massaroni View on GitHub
discussion enhancement feature-request review waiting for update
Dominant language
TypeScript
Stars
14.9k
Forks
1.1k
PR merge metrics
No merged PRs in 30d

Description

![A GIF or MEME to give some spice of the internet](http://www.cutecatgifs.com/wp-content/uploads/2013/03/focused.gif)

(I'm opening a new issue for this to start a conversation before submitting a pull request, so please let me know what you think.)

This adds new functionality to trainAsyc(), so that NodeJS users can utilize multiple gpus and cpus to train a single NeuralNetwork. This should significantly speed up training if you have a large neural net and/or a large training data set.

Is this a feature that we would want to merge into develop? [y/n]

### Code
This branch, based on master, has a working example: [massaroni/feature-parallel-training-m](https://github.com/massaroni/brain.js/tree/feature-parallel-training-m)
This other branch is mergeable into develop, but develop is too unstable at this point to demo the multithreaded training. [massaroni/feature-parallel-training](https://github.com/massaroni/brain.js/tree/feature-parallel-training)

See the example in [parallel-trainer-example.js](https://github.com/massaroni/brain.js/blob/feature-parallel-training-m/examples/parallel-trainer-example.js). It basically just shows that the algorithm does converge.
See the main functionality in [parallel-trainer.js](https://github.com/massaroni/brain.js/blob/feature-parallel-training/src/parallel-trainer.js)

### Documentation
`trainAsync()` in parallel mode, can train a single net on multiple threads. This should speed up training for large nets, large training sets, or both.

Train a NeuralNetwork on 3 cpu threads.
```javascript
const net = new brain.NeuralNetwork();
net
.trainAsync(data, {
parallel: {
threads: 3,
partitionSize: 1500, // optional. send a partition of 1500 items from the training set to each thread. Raise this number to get some overlap in the training data partitions.
epochs: 20000, // optional. limit each thread to 20,000 training runs
},
// ... and the usual training options
})
.then(res => {
// do something with my trained network
})
.catch(handleError);
```

Train a NeuralNetwork on 6 cpu threads and 2 GPU threads.
```javascript
const net = new brain.NeuralNetwork();
net
.trainAsync(data, {
parallel: {
threads: {
NeuralNetwork: 6,
NeuralNetworkGPU: 2
}
},
// ... and the usual training options
})
.then(res => {
// do something with my trained network
})
.catch(handleError);
```

Train a single NeuralNetwork on 6 cpu threads and 2 GPU threads, and send 10x more training data to the GPUs because they can run through it faster.
```javascript
const net = new brain.NeuralNetwork();
net
.trainAsync(data, {
parallel: {
threads: {
NeuralNetwork: {
threads: 6,
trainingDataSize: 2200
},
NeuralNetworkGPU: {
threads: 2,
trainingDataSize: 22000
}
}
},
// ... and the usual training options
})
.then(res => {
// do something with my trained network
})
.catch(handleError);
```

### Roadmap
* support all other neural net types
* web workers, for multithreaded training in the browser
* distributed training (multiple machines) (async SGD w/stale gradient handling?)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.