hpc / hpc/mpifileutils

dbcast Seg Fault

Open
#500 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
200
Forks
85
Avg merge
3d 21h
Merged PRs (30d)
2

Description

Running the dbcast command and I encounter a segfault; I'm using a main node to manage two worker nodes, and the crash has so far always occurred on the second worker. The first worker is successful.

Command from main node is
mpirun -np 128 --hostfile hostfile --mca btl_tcp_if_exclude virbr0,lo,ib0 dbcast /home/mpiuser/cloud/128-files.0.0 /home/mpiuser/dbcastfile.0.0

Every process backtrace has the same pattern; looks like some kind of crash in shared memory. I can try to recompile with debug symbols. Will post a response as I gather more data.

Oct  5 16:14:50 localhost systemd-coredump[89949]: Process 89887 (dbcast) of user 1000 dumped core.

Stack trace of thread 89887:
0x00007f0b1d841187 strmap_unset (libmfu.so.3.0.0)#012#1  
0x00007f0b1d841421 strmap_unsetf (libmfu.so.3.0.0)#012#2  
0x0000000000402b04 GCS_Shmem_free (dbcast)#012#3  
0x00000000004060c7 main (dbcast)#012#4  
0x00007f0b1c4d1493 __libc_start_main (libc.so.6)#012#5  
0x000000000040226e _start (dbcast)#012#012

Stack trace of thread 89910:#012#0  
0x00007f0b1c59fa41 __poll (libc.so.6)#012#1  
0x00007f0b1bd73015 poll_dispatch (libopen-pal.so.40)#012#2  
0x00007f0b1bd6a5d9 opal_libevent2022_event_base_loop (libopen-pal.so.40)#012#3  
0x00007f0b1bd26e4e progress_engine (libopen-pal.so.40)#012#4  
0x00007f0b1b2af14a start_thread (libpthread.so.0)#012#5  
0x00007f0b1c5aadc3 __clone (libc.so.6)#012#012

Stack trace of thread 89915:#012#0  
0x00007f0b1c5ab0f7 epoll_wait (libc.so.6)#012#1  
0x00007f0b1bd6653d epoll_dispatch (libopen-pal.so.40)#012#2  
0x00007f0b1bd6a5d9 opal_libevent2022_event_base_loop (libopen-pal.so.40)#012#3  
0x00007f0b13d646be progress_engine (mca_pmix_pmix3x.so)#012#4  
0x00007f0b1b2af14a start_thread (libpthread.so.0)#012#5  
0x00007f0b1c5aadc3 __clone (libc.so.6)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the dbcast command with its hostfile and MPI settings, then inspect the GCS_Shmem_free path and the strmap_unset/strmap_unsetf frames in libmfu. Recompile with debug symbols as suggested and compare behavior on the second worker. Done means the segfault cause is isolated and the fix is verified with this two-worker run.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
distributed-systems, hpc
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.