Batched input issue for nodes using auxiliary memory
- Dominant language
- C++
- Stars
- 3.4k
- Forks
- 701
- PR merge metrics
- No merged PRs in 30d
Description
Hi,
Nodes that do not explicitly support batched inputs should not be automatically batched by sequentially feeding each batch to the forward method: if the node stores values in its auxiliary memory so they can be used during backward, then it will results in an incorrect gradient.
The problem comes from this part of the code: https://github.com/clab/dynet/blob/946200c03521d06b620c473bcd64321d557e2451/dynet/dynet.cc#L48
Note that I don't think that checking if the nodes use auxiliary memory would be a good workaround: the problem can also happens if temporary data is stored in an instance variable (it is probably a bad design to not use auxiliary memory managed by dynet, but you never know).
**Example with sparsemax**
During foward, the node stores a float here: https://github.com/clab/dynet/blob/946200c03521d06b620c473bcd64321d557e2451/dynet/nodes-softmaxes.cc#L335
If the input is a batched input, this value will be overwritten for each batch, so its value will be the one of the last batch.
This value is used to compute the gradient during backward here: https://github.com/clab/dynet/blob/946200c03521d06b620c473bcd64321d557e2451/dynet/nodes-softmaxes.cc#L352
So only the last batch will have a correct gradient.
Here is a proof-of-concept: https://gist.github.com/FilippoC/9172a0513d85629df2a24ad4a98e6311
If I split the batch manually and create a different sparsemax node for each batch, the gradient checking succeeds. If I give the batch expression to a single sparsemax node, it fails.
```
0 [filippo dynet]% g++-7 sparsemax.cpp -std=c++11 -Wall -ldynet -Wfatal-errors
0 [filippo dynet]% ./a.out
[dynet] random seed: 3914237081
[dynet] allocating memory: 512MB
[dynet] memory allocation done.
Automatic batching: fail
Manual batching: pass
```
Best,
Caio
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the automatic batching logic in dynet/dynet.cc around line 48, then inspect the sparsemax forward and backward code in nodes-softmaxes.cc around lines 335 and 352. Run the linked sparsemax proof of concept and compare automatic with manual batching. Done means unsupported batched nodes no longer produce incorrect gradients, including the sparsemax case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100