clab / clab/dynet

Batched input issue for nodes using auxiliary memory

Open
#1,490 2 comments 0 reactions 0 assignees View on GitHub
major bug
Dominant language
C++
Stars
3.4k
Forks
701
PR merge metrics
No merged PRs in 30d

Description

Hi,

Nodes that do not explicitly support batched inputs should not be automatically batched by sequentially feeding each batch to the forward method: if the node stores values in its auxiliary memory so they can be used during backward, then it will results in an incorrect gradient.

The problem comes from this part of the code: https://github.com/clab/dynet/blob/946200c03521d06b620c473bcd64321d557e2451/dynet/dynet.cc#L48

Note that I don't think that checking if the nodes use auxiliary memory would be a good workaround: the problem can also happens if temporary data is stored in an instance variable (it is probably a bad design to not use auxiliary memory managed by dynet, but you never know).

**Example with sparsemax**

During foward, the node stores a float here: https://github.com/clab/dynet/blob/946200c03521d06b620c473bcd64321d557e2451/dynet/nodes-softmaxes.cc#L335
If the input is a batched input, this value will be overwritten for each batch, so its value will be the one of the last batch.
This value is used to compute the gradient during backward here: https://github.com/clab/dynet/blob/946200c03521d06b620c473bcd64321d557e2451/dynet/nodes-softmaxes.cc#L352
So only the last batch will have a correct gradient.

Here is a proof-of-concept: https://gist.github.com/FilippoC/9172a0513d85629df2a24ad4a98e6311
If I split the batch manually and create a different sparsemax node for each batch, the gradient checking succeeds. If I give the batch expression to a single sparsemax node, it fails.

```
0 [filippo dynet]% g++-7 sparsemax.cpp -std=c++11 -Wall -ldynet -Wfatal-errors
0 [filippo dynet]% ./a.out
[dynet] random seed: 3914237081
[dynet] allocating memory: 512MB
[dynet] memory allocation done.
Automatic batching: fail
Manual batching: pass
```

Best,
Caio

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the automatic batching logic in dynet/dynet.cc around line 48, then inspect the sparsemax forward and backward code in nodes-softmaxes.cc around lines 335 and 352. Run the linked sparsemax proof of concept and compare automatic with manual batching. Done means unsupported batched nodes no longer produce incorrect gradients, including the sparsemax case.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.