Diagnosing `NODE_FAILURE` when using NextFlow on Cheaha
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 24
- Forks
- 15
- Avg merge
- 10d 3h
- Merged PRs (30d)
- 1
Description
What would you like to see added?
If the following conditions are true, then consider that one or more NextFlow tasks may have insufficient memory allocated. Assume $jobid is the Slurm Job ID for the relevant NextFlow task.
sacct -j $jobid -X -o jobid,stateshowsNODE_FAILURE- The file
.exitcodedoes not exist in the NextFlow task's working directory. Working directory here refers to the NextFlow concept. - No other helpful messages showing a more detailed description of the cause of the error, a more specific error name, or an exit code.
- Tasks fail intermittently (not necessary, but increases likelihood)
When we have encountered researcher workflows where the above are true, the cause of the error has invariably been due to an "Out of Memory" (OOM) event.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Use the repository's MkDocs documentation structure to locate the relevant NextFlow or Slurm page. Add the stated NODE_FAILURE, missing .exitcode, absent detailed error, and intermittent-failure indicators, along with the OOM interpretation. Done means researchers can use the page to recognize this failure pattern.
Written by the indexing model from the issue text.
Assessment
- Domain
- documentation, hpc
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100