clab / clab/dynet

Highway RNN builder

Open
#1,198 8 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
C++
Stars
3.4k
Forks
701
PR merge metrics
No merged PRs in 30d

Description

A highway LSTM ([Srivastava et al., 2015](https://papers.nips.cc/paper/5850-training-very-deep-networks.pdf); [Zhang et al., 2016](https://groups.csail.mit.edu/sls/publications/2016/YuZhang3_ICASSP-16.pdf); [He et al., 2017](http://aclweb.org/anthology/P/P17/P17-1044.pdf)) is similar to a deep BiLSTM, but each layer's output is linearly combined with its input before being passed on to the next layer, where the combination is determined by a gate depending on the previous time step and the previous layer's output. This alleviates vanishing gradients along the layers.
A highway RNN builder shouldn't be much harder than the existing BiRNN builder to implement.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the cited highway-network papers and locating the existing BiRNN builder in the DyNet codebase. Compare its interface and behavior with the requested highway LSTM design; done means a highway RNN builder is implemented with the described gated layer connections and validated against the existing builder's conventions.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.