<LMON BE API> (ERROR): read_lmonp return a neg value
Nobody has claimed this yet.
- Dominant language
- C
- Stars
- 70
- Forks
- 25
- PR merge metrics
- No merged PRs in 30d
Description
Hi,
I did some tests with the latest stat-cl.
It works file on runs with 45 nodes x 16 MPI
but it fails with the error from bellow on a run with 880 nodes x 8 MPI
```
[atosla@ac1-1001 ~]$ stat-cl `pgrep -o srun` 2>~/stat-cl.err
STAT started at 2022-07-08-13:44:15
Attaching to job launcher (null):690121 and launching tool daemons...
(INFO): Just continued the RM process out of the first trap
(ERROR): read_lmonp return a neg value
```
In stat-cl.err I see the following
```
graph is not connected, found 0 potential roots
STAT returned error type STAT_MRNET_ERROR: MRNet reported a Network error with message: MRNet: Topology error: file has incorrect format
STAT returned error type STAT_MRNET_ERROR: Failed to launch MRNet tree()
ac3-2041.bullx: STAT returned error type STAT_LMON_ERROR: Failed to receive data from FE
ac3-2041.bullx: STAT returned error type STAT_LMON_ERROR: Failed to connect BE
ac3-1087.bullx: STAT returned error type STAT_LMON_ERROR: Failed to receive data from FE
ac3-1087.bullx: STAT returned error type STAT_LMON_ERROR: Failed to connect BE
ac1-4064.bullx: STAT returned error type STAT_LMON_ERROR: Failed to receive data from FE
ac1-4064.bullx: STAT returned error type STAT_LMON_ERROR: Failed to connect BE
ac3-2012.bullx: STAT returned error type STAT_LMON_ERROR: Failed to receive data from FE
ac3-2012.bullx: STAT returned error type STAT_LMON_ERROR: Failed to connect BE
ac1-4008.bullx: STAT returned error type STAT_LMON_ERROR: Failed to receive data from FE
ac1-4008.bullx: STAT returned error type STAT_LMON_ERROR: Failed to connect BE
ac3-2013.bullx: STAT returned error type STAT_LMON_ERROR: Failed to receive data from FE
ac3-2013.bullx: STAT returned error type STAT_LMON_ERROR: Failed to connect BE
ac3-3013.bullx: STAT returned error type STAT_LMON_ERROR: Failed to receive data from FE
ac3-3013.bullx: STAT returned error type STAT_LMON_ERROR: Failed to connect BE
ac3-2033.bullx: STAT returned error type STAT_LMON_ERROR: Failed to receive data from FE
ac3-2033.bullx: STAT returned error type STAT_LMON_ERROR: Failed to connect BE
...
```
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the failure with stat-cl on the reported 880-node, 8-MPI-per-node run and compare it with the working 45-node case. Start from the LMON BE API read_lmonp error and the reported MRNet topology error; done means the large run launches successfully without the disconnected-graph or failed-connection errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c
- Domain
- distributed-systems, tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100