jobs stuck in PENDING state but they've actually completed or are currently running
- Dominant language
- Python
- Stars
- 161
- Forks
- 49
- PR merge metrics
- No merged PRs in 30d
Description
I've encountered the confusing situation of having the conductor still running as a background process, my jobs have all completed successfully, but running `maestro status` lists the jobs that were run via SLURM as still "PENDING".
Is there any way to figure out what has gone wrong, or otherwise reset the conductor process?
Running `ps aus | grep $USER`, I see:
```
login4.stampede3(626)$ ps aux | grep $USER
bwibking 1549018 0.0 0.0 20820 11936 ? Ss Apr30 0:09 /usr/lib/systemd/systemd --user
bwibking 1549019 0.0 0.0 202568 6848 ? S Apr30 0:00 (sd-pam)
bwibking 1725176 0.0 0.0 8436 3692 ? S Apr30 0:00 /bin/sh -c nohup conductor -t 60 -d 2 /scratch/02661/bwibking/precipitator-paper/outputs/medres_compressive_20240430-160408 > /scratch/02661/bwibking/precipitator-paper/outputs/medres_compressive_20240430-160408/medres_compressive.txt 2>&1
bwibking 1725177 0.0 0.0 325552 67876 ? S Apr30 0:32 /scratch/projects/compilers/intel24.0/oneapi/intelpython/python3.9/bin/python3.9 /home1/02661/bwibking/.local/bin/conductor -t 60 -d 2 /scratch/02661/bwibking/precipitator-paper/outputs/medres_compressive_20240430-160408
root 3152422 0.0 0.0 39932 11976 ? Ss 13:31 0:00 sshd: bwibking [priv]
bwibking 3152677 0.0 0.0 40240 7472 ? S 13:31 0:00 sshd: bwibking@pts/122
bwibking 3152678 0.0 0.0 17920 6116 pts/122 Ss 13:31 0:00 -bash
root 3163591 0.0 0.0 39932 12044 ? Ss 13:43 0:00 sshd: bwibking [priv]
bwibking 3163629 0.0 0.0 40076 7236 ? S 13:43 0:00 sshd: bwibking@notty
root 3168472 0.0 0.0 39932 12020 ? Ss 13:48 0:00 sshd: bwibking [priv]
bwibking 3168505 0.0 0.0 40080 7208 ? S 13:48 0:00 sshd: bwibking@notty
bwibking 3174162 0.0 0.0 20408 4220 pts/122 R+ 13:54 0:00 ps aux
bwibking 3174163 0.0 0.0 7592 2836 pts/122 S+ 13:54 0:00 grep --color=auto bwibking
```
Partial output from `maestro status`:
```
run-sim_amp.0.0002222222222222222.f_sol.0.0.tc_tff.1.333521432163324 3 run-sim/amp.0.0002222222222222222.f_sol.0.0.tc_tff.1.333521432163324 PENDING --:--:-- --:--:-- -- 2024-04-30 16:06:52 -- 0
run-sim_amp.0.0003888888888888888.f_sol.0.0.tc_tff.4.216965034285822 3 run-sim/amp.0.0003888888888888888.f_sol.0.0.tc_tff.4.216965034285822 PENDING --:--:-- --:--:-- -- 2024-04-30 16:06:54 -- 0
run-sim_amp.0.00011111111111111112.f_sol.0.0.tc_tff.2.371373705661655 3 run-sim/amp.0.00011111111111111112.f_sol.0.0.tc_tff.2.371373705661655 PENDING --:--:-- --:--:-- -- 2024-04-30 16:06:55 -- 0
run-sim_amp.0.0002777777777777778.f_sol.0.0.tc_tff.7.498942093324558 3 run-sim/amp.0.0002777777777777778.f_sol.0.0.tc_tff.7.498942093324558 PENDING --:--:-- --:--:-- -- 2024-04-30 16:06:57 -- 0
run-sim_amp.0.00044444444444444447.f_sol.0.0.tc_tff.1.1547819846894583 3 run-sim/amp.0.00044444444444444447.f_sol.0.0.tc_tff.1.1547819846894583 PENDING --:--:-- --:--:-- -- 2024-04-30 16:06:58 -- 0
run-sim_amp.1.8518518518518515e-05.f_sol.0.0.tc_tff.3.651741272548377 3 run-sim/amp.1.8518518518518515e-05.f_sol.0.0.tc_tff.3.651741272548377 PENDING --:--:-- --:--:-- -- 2024-04-30 16:07:00 -- 0
run-sim_amp.0.00018518518518518518.f_sol.0.0.tc_tff.2.0535250264571463 3 run-sim/amp.0.00018518518518518518.f_sol.0.0.tc_tff.2.0535250264571463 PENDING --:--:-- --:--:-- -- 2024-04-30 16:07:02 -- 0
run-sim_amp.0.0003518518518518519.f_sol.0.0.tc_tff.6.493816315762113 3 run-sim/amp.0.0003518518518518519.f_sol.0.0.tc_tff.6.493816315762113 PENDING --:--:-- --:--:-- -- 2024-04-30 16:07:04 -- 0
run-sim_amp.7.407407407407406e-05.f_sol.0.0.tc_tff.1.539926526059492 3 run-sim/amp.7.407407407407406e-05.f_sol.0.0.tc_tff.1.539926526059492 PENDING --:--:-- --:--:-- -- 2024-04-30 16:07:06 -- 0
run-sim_amp.0.00024074074074074072.f_sol.0.0.tc_tff.4.869675251658631 3 run-sim/amp.0.00024074074074074072.f_sol.0.0.tc_tff.4.869675251658631 PENDING --:--:-- --:--:-- -- 2024-04-30 16:07:08 -- 0
```
Contributor guide
Research direction
No source files or tests are identified. Start by reproducing the SLURM workflow with the `conductor` background process and compare scheduler state with `maestro status`; trace how completed or running jobs remain PENDING. Done means the status output reflects actual job state and provides a reliable recovery or reset path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100