clab / clab/dynet

Segmentation fault when using auto batching

Open
#1,467 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
3.4k
Forks
701
PR merge metrics
No merged PRs in 30d

Description

Hi,

I encounter segmentation fault when enabling **--dynet-autobatch** flag on both CPU and GPU. The error message is as follow:
```
[dynet] initializing CUDA
[dynet] CUDA driver/runtime versions are 9.1/9.1
Request for 1 GPU ...
[dynet] Device Number: 0
[dynet] Device name: GeForce GTX 1080 Ti
[dynet] Memory Clock Rate (KHz): 5505000
[dynet] Memory Bus Width (bits): 352
[dynet] Peak Memory Bandwidth (GB/s): 484.44
[dynet] Memory Free (GB): 11.5442/11.7215
[dynet]
[dynet] Device(s) selected: 0
[dynet] random seed: 3273524966
[dynet] using autobatching
[dynet] allocating memory: 10000MB
[dynet] memory allocation done.
/var/spool/slurmd/job97399/slurm_script: line 10: 13286 Segmentation fault python ./inference/context_edl/edl_model_debug.py --dynet-gpus 1 --dynet-mem 10000 --dynet-autobatch 1
```

This fault doesn’t occur as soon as I start training my model, it often occurs after I train ~2000 samples.

BTW, I found all flags in the command line (e.g. enable dynet-gpu, dynet-autobatch) will be ignored if I add the following lines to my code:
```python
import dynet_config
dynet_config.set(random_seed=2407925251) # set some values
```
It will be OK if I just import dynet_config without setting any value.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the failure with inference/context_edl/edl_model_debug.py using --dynet-autobatch, then compare runs with and without dynet_config.set(random_seed=2407925251). Check whether the command-line flags are applied and whether the process still segfaults after roughly 2,000 samples. Done means autobatching runs without a segmentation fault and configuration flags are honored.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
backend, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.