LSTMs cannot use different batch sizes in the same computation graph
- Dominant language
- C++
- Stars
- 3.4k
- Forks
- 701
- PR merge metrics
- No merged PRs in 30d
Description
Now that DyNet has switched to variational dropout, a dropout mask needs to be stored so that the same mask can be applied across all time steps. Unfortunately this means that an LSTM cannot be used with different batch sizes in the same computation graph, since the initial mask will have the wrong shape for later batches.
Here's a minimal example demonstrating the problem:
```python
import dynet as dy
model = dy.Model()
lstm = dy.BiRNNBuilder(1, 5, 20, model, dy.VanillaLSTMBuilder)
lstm.set_dropout(0.5)
lstm.transduce([dy.zeros(5, batch_size=2)])
# Now the dropout mask has batch size 2, so it cannot be applied to other batch sizes
lstm.transduce([dy.zeros(5, batch_size=3)])
```
This results in the error:
```
ValueError: CwiseMultiply: batch size must match or equal 1
```
Although the above code will run if both batch sizes are the same, we may not want to apply the same dropout mask to different examples. Additionally, if the first batch size is 2 but the second batch size is 1, broadcasting will result in an output with batch size 2 for the latter result, which is incorrect behavior.
The use case here is that I would like to run an LSTM over every fixed-size window within a sentence, then do this for multiple sentences within a batch. The batch size in each call is the length of the sentence.
Is there any way around this? Perhaps a new mask could be generated for each new initial state within a given computation graph?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.