clab / clab/dynet

examples/train_rnnlm-mp crashes with SIGABRT

Open
#879 2 comments 0 reactions 0 assignees View on GitHub
moderate bug
Dominant language
C++
Stars
3.4k
Forks
701
PR merge metrics
No merged PRs in 30d

Description

### Summary
I ran examples/train_rnnlm-mp on my macOS (10.12.6), and then it crashed with SIGABRT.

```
$ ./examples/train_rnnlm-mp 1 /tmp/ptb-mikolov/train.txt /tmp/ptb-mikolov/valid.txt
[dynet] random seed: 601977149
[dynet] allocating memory: 512MB
[dynet] memory allocation done.
libc++abi.dylib: terminating with uncaught exception of type std::invalid_argument: Out-of-bounds attempt to access index 5773 for LookupParameter of size 5500
```

### Environment

- macOS (10.12.6)
- clang version: Apple LLVM version 8.0.0 (clang-800.0.42.1)
- Boost version: 1.64.0

### Steps to reproduce
```
$ git clone --depth 1 git@github.com:clab/dynet.git
$ cd dynet
$ mkdir build
$ cd build
$ cmake .. -DEIGEN3_INCLUDE_DIR=/usr/local/include/eigen3 -DENABLE_CPP_EXAMPLES=ON -DENABLE_BOOST=ON -G Ninja
$ ninja
$ ./examples/train_rnnlm-mp 1 /tmp/ptb-mikolov/train.txt /tmp/ptb-mikolov/valid.txt
[dynet] random seed: 601977149
[dynet] allocating memory: 512MB
[dynet] memory allocation done.
libc++abi.dylib: terminating with uncaught exception of type std::invalid_argument: Out-of-bounds attempt to access index 5773 for LookupParameter of size 5500
```

Here is the generated crash log found at `~/Library/Logs/DiagnosticReports/`:

```
Process: train_rnnlm-mp [81488]
Path: /Users/USER/*/train_rnnlm-mp
Identifier: train_rnnlm-mp
Version: 0
Code Type: X86-64 (Native)
Parent Process: train_rnnlm-mp [81486]
Responsible: train_rnnlm-mp [81488]
User ID: 501

Date/Time: 2017-09-08 23:33:13.443 +0900
OS Version: Mac OS X 10.12.6 (16G29)
Report Version: 12
Anonymous UUID: B8964C10-E7AF-79E7-46FE-60F250ACDDA6

Sleep/Wake UUID: 788DEE93-EDB5-4B0E-85EA-D98B3818D2B1

Time Awake Since Boot: 320000 seconds
Time Since Wake: 1900 seconds

System Integrity Protection: enabled

Crashed Thread: 0 Dispatch queue: com.apple.main-thread

Exception Type: EXC_CRASH (SIGABRT)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Exception Note: EXC_CORPSE_NOTIFY

Application Specific Information:
terminating with uncaught exception of type std::invalid_argument: Out-of-bounds attempt to access index 5773 for LookupParameter of size 5500
crashed on child side of fork pre-exec

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0 libsystem_kernel.dylib 0x00007fffba753d42 __pthread_kill + 10
1 libsystem_pthread.dylib 0x00007fffba841457 pthread_kill + 90
2 libsystem_c.dylib 0x00007fffba6b9420 abort + 129
3 libc++abi.dylib 0x00007fffb920c94a abort_message + 266
4 libc++abi.dylib 0x00007fffb9231c17 default_terminate_handler() + 243
5 libobjc.A.dylib 0x00007fffb9d41713 _objc_terminate() + 124
6 libc++abi.dylib 0x00007fffb922ed49 std::__terminate(void (*)()) + 8
7 libc++abi.dylib 0x00007fffb922e7be __cxa_throw + 121
8 libdynet.dylib 0x000000010c978922 void dynet::LookupNode::forward_dev_impl(dynet::Device_CPU const&, std::__1::vector > const&, dynet::Tensor&) const + 2930 (param-nodes.cc:307)
9 libdynet.dylib 0x000000010c71b7d2 dynet::SimpleExecutionEngine::incremental_forward(unsigned int) + 1362 (exec.cc:87)
10 libdynet.dylib 0x000000010c71afdf dynet::SimpleExecutionEngine::forward(unsigned int) + 31 (exec.cc:47)
11 libdynet.dylib 0x000000010c719391 dynet::ComputationGraph::forward(dynet::Expression const&) + 17 (dynet.cc:375)
12 train_rnnlm-mp 0x000000010c6e1cce Learner > >::LearnFromDatum(std::__1::vector > const&, bool) + 78 (train_rnnlm-mp.cc:47)
13 train_rnnlm-mp 0x000000010c6e30ba int dynet::mp::run_child >, float>(unsigned int, dynet::mp::ILearner >, float>*, dynet::Trainer*, std::__1::vector >&, std::__1::vector >, std::__1::allocator > > > const&, std::__1::vector >, std::__1::allocator > > > const&) + 474 (mp.h:248)
14 train_rnnlm-mp 0x000000010c6ddcf5 void dynet::mp::run_multi_process >, float>(unsigned int, dynet::mp::ILearner >, float>*, dynet::Trainer*, std::__1::vector >, std::__1::allocator > > > const&, std::__1::vector >, std::__1::allocator > > > const&, unsigned int, unsigned int, unsigned int) + 501 (mp.h:297)
15 train_rnnlm-mp 0x000000010c6dd86c main + 636 (rnnlm.h:21)
16 libdyld.dylib 0x00007fffba625235 start + 1

Thread 0 crashed with X86 Thread State (64-bit):
rax: 0x0000000000000000 rbx: 0x0000000000000006 rcx: 0x00007fff53524268 rdx: 0x0000000000000000
rdi: 0x0000000000000203 rsi: 0x0000000000000006 rbp: 0x00007fff53524290 rsp: 0x00007fff53524268
r8: 0x00007fffba6e2650 r9: 0x00007fffba6a4a50 r10: 0x0000000008000000 r11: 0x0000000000000206
r12: 0x00007fff535243f0 r13: 0x0000000000000030 r14: 0x00007fffc35393c0 r15: 0x0000000000000008
rip: 0x00007fffba753d42 rfl: 0x0000000000000206 cr2: 0x00007fffb920f1b3

Logical CPU: 0
Error Code: 0x02000148
Trap Number: 133

Binary Images:
0x10c6da000 - 0x10c6ecfef +train_rnnlm-mp (0) <3D1E5572-44C1-329C-86CD-3C7B890A2C7D> /Users/USER/*/train_rnnlm-mp
0x10c702000 - 0x10c9e8ff7 +libdynet.dylib (0) /Users/USER/*/libdynet.dylib
0x10cb69000 - 0x10cb9dfff +libboost_program_options-mt.dylib (0) /usr/local/opt/boost/lib/libboost_program_options-mt.dylib
0x10cbfa000 - 0x10cc9aff7 +libboost_regex-mt.dylib (0) /usr/local/opt/boost/lib/libboost_regex-mt.dylib
0x10cd27000 - 0x10cd4aff7 +libboost_serialization-mt.dylib (0) <5F4D0E28-7CF7-380F-A423-CB8BEA3CD136> /usr/local/opt/boost/lib/libboost_serialization-mt.dylib
0x1135db000 - 0x113618dc7 dyld (433.5) <322C06B7-8878-311D-888C-C8FD2CA96FF3> /usr/lib/dyld
0x7fffb907b000 - 0x7fffb907cffb libSystem.B.dylib (1238.60.2) /usr/lib/libSystem.B.dylib
0x7fffb91b5000 - 0x7fffb920bff7 libc++.1.dylib (307.5) <0B43BB5D-E6EB-3464-8DE9-B41AC8ED9D1C> /usr/lib/libc++.1.dylib
0x7fffb920c000 - 0x7fffb9235ff7 libc++abi.dylib (307.4) /usr/lib/libc++abi.dylib
0x7fffb9d2a000 - 0x7fffba0fc047 libobjc.A.dylib (709.1) <70614861-0340-32E2-85ED-FE65759CDFFA> /usr/lib/libobjc.A.dylib
0x7fffba549000 - 0x7fffba54dff7 libcache.dylib (79) <093A4DAB-8385-3D47-A350-E20CB7CCF7BF> /usr/lib/system/libcache.dylib
0x7fffba54e000 - 0x7fffba558fff libcommonCrypto.dylib (60092.50.5) <8A64D1B0-C70E-385C-92F0-E669079FDA90> /usr/lib/system/libcommonCrypto.dylib
0x7fffba559000 - 0x7fffba560fff libcompiler_rt.dylib (62) <55D47421-772A-32AB-B529-1A46C2F43B4D> /usr/lib/system/libcompiler_rt.dylib
0x7fffba561000 - 0x7fffba569fff libcopyfile.dylib (138) <819BEA3C-DF11-3E3D-A1A1-5A51C5BF1961> /usr/lib/system/libcopyfile.dylib
0x7fffba56a000 - 0x7fffba5edfdf libcorecrypto.dylib (442.50.19) <65D7165E-2E71-335D-A2D6-33F78E2DF0C1> /usr/lib/system/libcorecrypto.dylib
0x7fffba5ee000 - 0x7fffba61ffff libdispatch.dylib (703.50.37) <6582BAD6-ED27-3B30-B620-90B1C5A4AE3C> /usr/lib/system/libdispatch.dylib
0x7fffba620000 - 0x7fffba625ffb libdyld.dylib (433.5) <9B2AC56D-107C-3541-A127-9094A751F2C9> /usr/lib/system/libdyld.dylib
0x7fffba626000 - 0x7fffba626ffb libkeymgr.dylib (28) <7AA011A9-DC21-3488-BF73-3B5B14D1FDD6> /usr/lib/system/libkeymgr.dylib
0x7fffba634000 - 0x7fffba634fff liblaunch.dylib (972.70.1) /usr/lib/system/liblaunch.dylib
0x7fffba635000 - 0x7fffba63aff3 libmacho.dylib (898) <17D5D855-F6C3-3B04-B680-E9BF02EF8AED> /usr/lib/system/libmacho.dylib
0x7fffba63b000 - 0x7fffba63dff3 libquarantine.dylib (85.50.1) <12448CC2-378E-35F3-BE33-9DC395A5B970> /usr/lib/system/libquarantine.dylib
0x7fffba63e000 - 0x7fffba63fffb libremovefile.dylib (45) <38D4CB9C-10CD-30D3-8B7B-A515EC75FE85> /usr/lib/system/libremovefile.dylib
0x7fffba640000 - 0x7fffba658ff7 libsystem_asl.dylib (349.50.5) <096E4228-3B7C-30A6-8B13-EC909A64499A> /usr/lib/system/libsystem_asl.dylib
0x7fffba659000 - 0x7fffba659ff7 libsystem_blocks.dylib (67) <10DC5404-73AB-35B3-A277-A8AFECB476EB> /usr/lib/system/libsystem_blocks.dylib
0x7fffba65a000 - 0x7fffba6e7fef libsystem_c.dylib (1158.50.2) /usr/lib/system/libsystem_c.dylib
0x7fffba6e8000 - 0x7fffba6ebffb libsystem_configuration.dylib (888.60.2) /usr/lib/system/libsystem_configuration.dylib
0x7fffba6ec000 - 0x7fffba6effff libsystem_coreservices.dylib (41.4) <7D26DE79-B424-3450-85E1-F7FAB32714AB> /usr/lib/system/libsystem_coreservices.dylib
0x7fffba6f0000 - 0x7fffba708fff libsystem_coretls.dylib (121.50.4) /usr/lib/system/libsystem_coretls.dylib
0x7fffba709000 - 0x7fffba70ffff libsystem_dnssd.dylib (765.50.9) /usr/lib/system/libsystem_dnssd.dylib
0x7fffba710000 - 0x7fffba739ff7 libsystem_info.dylib (503.50.4) <611DB84C-BF70-3F92-8702-B9F28A900920> /usr/lib/system/libsystem_info.dylib
0x7fffba73a000 - 0x7fffba75cff7 libsystem_kernel.dylib (3789.70.16) <34B1F16C-BC9C-3C5F-9045-0CAE91CB5914> /usr/lib/system/libsystem_kernel.dylib
0x7fffba75d000 - 0x7fffba7a4fe7 libsystem_m.dylib (3121.6) <86D499B5-BBDC-3D3B-8A4E-97AE8E6672A4> /usr/lib/system/libsystem_m.dylib
0x7fffba7a5000 - 0x7fffba7c3ff7 libsystem_malloc.dylib (116.50.8) /usr/lib/system/libsystem_malloc.dylib
0x7fffba7c4000 - 0x7fffba81dffb libsystem_network.dylib (856.60.1) <369D0221-56CA-3C3E-9EDE-94B41CAE77B7> /usr/lib/system/libsystem_network.dylib
0x7fffba81e000 - 0x7fffba827ff3 libsystem_networkextension.dylib (563.60.2) /usr/lib/system/libsystem_networkextension.dylib
0x7fffba828000 - 0x7fffba831ff3 libsystem_notify.dylib (165.20.1) /usr/lib/system/libsystem_notify.dylib
0x7fffba832000 - 0x7fffba83afe7 libsystem_platform.dylib (126.50.8) <897462FD-B318-321B-A554-E61982630F7E> /usr/lib/system/libsystem_platform.dylib
0x7fffba83b000 - 0x7fffba845ff7 libsystem_pthread.dylib (218.60.3) /usr/lib/system/libsystem_pthread.dylib
0x7fffba846000 - 0x7fffba849ff7 libsystem_sandbox.dylib (592.70.1) <4B92EC49-ACD0-36AE-B07A-A2B8152EAF9D> /usr/lib/system/libsystem_sandbox.dylib
0x7fffba84a000 - 0x7fffba84bff3 libsystem_secinit.dylib (24.50.4) /usr/lib/system/libsystem_secinit.dylib
0x7fffba84c000 - 0x7fffba853ffb libsystem_symptoms.dylib (532.50.47) <3390E07C-C1CE-348F-ADBD-2C5440B45EAA> /usr/lib/system/libsystem_symptoms.dylib
0x7fffba854000 - 0x7fffba867ff7 libsystem_trace.dylib (518.70.1) /usr/lib/system/libsystem_trace.dylib
0x7fffba868000 - 0x7fffba86dffb libunwind.dylib (35.3) <3D50D8A8-C460-334D-A519-2DA841102C6B> /usr/lib/system/libunwind.dylib
0x7fffba86e000 - 0x7fffba897ff7 libxpc.dylib (972.70.1) /usr/lib/system/libxpc.dylib

External Modification Summary:
Calls made by other processes targeting this process:
task_for_pid: 0
thread_create: 0
thread_set_state: 0
Calls made by this process:
task_for_pid: 0
thread_create: 0
thread_set_state: 0
Calls made by all processes on this machine:
task_for_pid: 198850
thread_create: 0
thread_set_state: 6

VM Region Summary:
ReadOnly portion of Libraries: Total=128.7M resident=0K(0%) swapped_out_or_unallocated=128.7M(100%)
Writable regions: Total=556.9M written=0K(0%) resident=0K(0%) swapped_out=0K(0%) unallocated=556.9M(100%)

VIRTUAL REGION
REGION TYPE SIZE COUNT (non-coalesced)
=========== ======= =======
Kernel Alloc Once 8K 2
MALLOC 420.9M 13
MALLOC guard page 16K 4
STACK GUARD 56.0M 2
Stack 8192K 2
VM_ALLOCATE 128.0M 3
__DATA 1528K 45
__LINKEDIT 116.9M 8
__TEXT 11.8M 45
mapped file 316K 2
shared memory 8K 3
=========== ======= =======
TOTAL 743.4M 118
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.