llnl / llnl/Spindle

spindleRunBE() failure prevents log file generation

Open
#147 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
110
Forks
35
Avg merge
1d 16h
Merged PRs (30d)
2

Description

A proposed `ci.yml` refactoring splits flux-ubuntu tests into four separate jobs and runs them in parallel on separate nodes. Three jobs cover SPINDLE_DEBUG=[1-3] and the fourth does not set SPINDLE_DEBUG.

The run in question succeeded for SPINDLE_DEBUG=[1,3] and the no-debugging option, but failed on SPINDLE_DEBUG=2 with the error message:

```
Run docker exec node-1 bash -c 'cd Spindle-build/testsuite && SPINDLE_DEBUG=2 flux alloc --nodes=${workers} ./runTests --nodes=${workers} --tasks-per-node=3'
docker exec node-1 bash -c 'cd Spindle-build/testsuite && SPINDLE_DEBUG=2 flux alloc --nodes=${workers} ./runTests --nodes=${workers} --tasks-per-node=3'
shell: /usr/bin/bash -e {0}
Running: ./run_driver --dependency --push
60.237s: flux-shell[0]: stderr: spindleRunBE failed!
Error: The action 'Run spindle-flux-ubuntu-debug2 testsuite' has timed out after 7 minutes.
```

At that point the CI script should extract the log files, but based on the `ls` the script performs, no files were generated. Looking at the likely code path, `LOGGING_FINI` is not called during any of the error paths, so the log files were likely not flushed to disk before spindle exited.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with ci.yml and the spindleRunBE error path referenced in the issue. Trace where LOGGING_FINI is called during failures, then run the SPINDLE_DEBUG=2 flux-ubuntu tests to verify that log files are flushed and available for CI extraction when the tests time out.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, docker
Domain
hpc, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.