Highway RNN builder
- Dominant language
- C++
- Stars
- 3.4k
- Forks
- 701
- PR merge metrics
- No merged PRs in 30d
Description
A highway LSTM ([Srivastava et al., 2015](https://papers.nips.cc/paper/5850-training-very-deep-networks.pdf); [Zhang et al., 2016](https://groups.csail.mit.edu/sls/publications/2016/YuZhang3_ICASSP-16.pdf); [He et al., 2017](http://aclweb.org/anthology/P/P17/P17-1044.pdf)) is similar to a deep BiLSTM, but each layer's output is linearly combined with its input before being passed on to the next layer, where the combination is determined by a gate depending on the previous time step and the previous layer's output. This alleviates vanishing gradients along the layers.
A highway RNN builder shouldn't be much harder than the existing BiRNN builder to implement.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the cited highway-network papers and locating the existing BiRNN builder in the DyNet codebase. Compare its interface and behavior with the requested highway LSTM design; done means a highway RNN builder is implemented with the described gated layer connections and validated against the existing builder's conventions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100