clab / clab/dynet

Autobatch segmentation fault

Open
#1,559 2 comments 1 reaction 0 assignees View on GitHub
need more info
Dominant language
C++
Stars
3.4k
Forks
701
PR merge metrics
No merged PRs in 30d

Description

Hi,

I have a problem using autobatching (it does not happen without).

Here is gdb output
`
Thread 1 "HNMTrainer" received signal SIGSEGV, Segmentation fault.
0x00007ffff51bbb9a in dynet::BatchedExecutionEngine::backward (this=, from_where=45896, full=)
at /home/amm/pba44c/Users/amm/Source/Repos/HierarchicalNestedNER/dynet/dynet/exec.cc:1074
1074 Tensor my_ndEdf = *xs[ai];

(gdb) p ai
$4 = 3
(gdb) p xs
$5 = std::vector of length 3, capacity 118 = {0x7ffe9e879090, 0x7ffe9e8790d0, 0x7fff14aed140}

`

The other variables were optimized out, this happens after 51000 updates and if I change the batch size it can happen much latter.

Now I can put the execute the code to this point jumping over the previous updates anf get to this state quickly.

If needed I can compile dynet without optimizations and do some guided debugging.

Thanks

Afonso

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at dynet/exec.cc:1074 and inspect the autobatching path around BatchedExecutionEngine::backward using the reported gdb state. Reproduce the failure with autobatching enabled, ideally by jumping to the state after the earlier updates as described. Done means the reported segmentation fault no longer occurs under the reproducing workload.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.