llnl / llnl/Spindle

Spindle changes library search behavior, causes libmpi to not be found

Open
#110 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
110
Forks
35
Avg merge
1d 16h
Merged PRs (30d)
2

Description

In the Ubuntu 24.04 images used for CI, if MPICH is installed but `ldconfig` is not run to update the cache, MPI applications will run successfully without Spindle, but with Spindle at level medium or high, libmpi is not found:

```
./mpi-hello: error while loading shared libraries: libmpi.so.12: cannot open shared object file: No such file or directory
```

`LD_DEBUG=all` output without Spindle, `ldconfig` not run:

```
file=libmpi.so.12 [0]; needed by /shared/examples/./mpi-hello [0]
find library=libmpi.so.12 [0]; searching
search path=/usr/lib/glibc-hwcaps/x86-64-v3:/usr/lib/glibc-hwcaps/x86-64-v2:/usr/lib (system search path)
trying file=/usr/lib/glibc-hwcaps/x86-64-v3/libmpi.so.12
trying file=/usr/lib/glibc-hwcaps/x86-64-v2/libmpi.so.12
trying file=/usr/lib/libmpi.so.12

file=libmpi.so.12 [0]; generating link map
dynamic: 0x000077e828e7c8c0 base: 0x000077e8263d5000 size: 0x0000000002ea1d10
entry: 0x000077e8263d5000 phdr: 0x000077e8263d5040 phnum: 12
```

With Spindle, `ldconfig` run:

```
file=libmpi.so.12 [0]; needed by ./mpi-hello [0]
find library=libmpi.so.12 [0]; searching
search path=/usr/lib/glibc-hwcaps/x86-64-v3:/usr/lib/glibc-hwcaps/x86-64-v2:/usr/lib (system search path)
trying file=/usr/lib/glibc-hwcaps/x86-64-v3/libmpi.so.12
trying file=/__not_exglibc-hwcaps/x86-64-v2/libmpi.so.12
trying file=/__not_exlibmpi.so.12
search cache=/etc/ld.so.cache
trying file=/usr/lib/libmpi.so.12

file=libmpi.so.12 [0]; generating link map
dynamic: 0x00007a2974da58c0 base: 0x00007a29722fe000 size: 0x0000000002ea1d10
entry: 0x00007a29722fe000 phdr: 0x00007a29722fe040 phnum: 12
```

While libmpi.so.12 is found in the cache, this behavior is already different from the non-Spindle case, which finds libmpi.so.12 _prior_ to searching the cache.

With Spindle, `ldconfig` not run, failing case:

```
file=libmpi.so.12 [0]; needed by ./mpi-hello [0]
find library=libmpi.so.12 [0]; searching
search path=/usr/lib/glibc-hwcaps/x86-64-v3:/usr/lib/glibc-hwcaps/x86-64-v2:/usr/lib (system search path)
trying file=/usr/lib/glibc-hwcaps/x86-64-v3/libmpi.so.12
trying file=/__not_exglibc-hwcaps/x86-64-v2/libmpi.so.12
trying file=/__not_exlibmpi.so.12
search cache=/etc/ld.so.cache
search path=/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v3:/usr/lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v2:/usr/lib/x8-gnu:/lib/hwcaps/x86-64-v3:/lib/glibc-hwcaps/x86-64-v2:/lib (system search path)
trying file=/lib/x86_64-linux-gnu/libmpi.so.12
trying file=/usr/lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v3/libmpi.so.12
trying file=/__not_exists/4-linux-gnu/glibc-hwcaps/x86-64-v2/libmpi.so.12
trying file=/__not_exists/4-linux-gnu/libmpi.so.12
trying file=/lib/glibc-hwcaps/x86-64-v3/libmpi.so.12
trying file=/__noglibc-hwcaps/x86-64-v2/libmpi.so.12
trying file=/__nolibmpi.so.12

./mpi-hello: error while loading shared libraries: libmpi.so.12: cannot open shared object file: No such file or directory
```

Here, `/usr/lib` doesn't get searched at all.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the issue in the Ubuntu 24.04 CI environment with MPICH, comparing MPI loading with Spindle disabled and at medium or high level. Start with the `mpi-hello` executable and the supplied `LD_DEBUG=all` traces, especially the `/usr/lib` and `/etc/ld.so.cache` searches. Done means Spindle no longer prevents `libmpi.so.12` from being found when `ldconfig` has not been run.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, linux, ubuntu
Domain
hpc, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.