Process management bug in RQD doesn't clean inactive or de-parented processes and causes "wedged" machines.

After analysing some wedged machines over the past week, I confirm the issue to be process management by RQD. Namely:

1. Management of crashed processes. A situation can arise where a certain process in the process chain that is started by RQD dies but the rest of the chain is still alive. RQD doesn't detect such a scenario and waits forever, not releasing the cores.
2. Management of output. This is a more tricky situation where a process in the chain writes to stdout/stderr but the parent process doesn't consume from the pipe, leading to a blocking write (pipe_w in system parlance).

Sadly, in both cases, trying to "reboot on idle" doesn't work because RQD thinks that there is still work being done.

--> Here is an illustration, taken directly from one of the wedged machines, of CASE 1.

```
python─┬─time
└─13*[{python}]
```

In this case, the "time" command is waiting for the process to finish (all RQD prcoesses are started with time "job command line"), but they all crashed but ONE, namely checkpoint_watch_adaptive, which we don't see in this graph above as it has been reparented to PID 1 by the operating system:

4 S ???? 259447 1 0 90 10 - 250378 futex_ Apr28 ? 00:00:23 python checkpoint_watch_adaptie ...

So we have a never ending process and RQD will never be "idle". The process will never release the allocated cores.

--> Here is an illustration, taken directly from one of the wedged machines, of CASE 2.

```
python─┬─2*[time───su───tcsh───sh───spk_wrap_frame───spawn─┬─pycuerun───python───frame_script]
│ └─6*[{spawn}]]
├─4*[time───su───tcsh───sh───spk_wrap_frame───spawn─┬─pycuerun───python───frame_script───3*[{frame_script}]]
│ └─6*[{spawn}]]
├─30*[time───su───tcsh───sh───spk_wrap_frame]
├─2*[time───su───tcsh───sh───spk_wrap_frame───spawn─┬─pycuerun───python]
│ └─6*[{spawn}]]
├─time───su───tcsh───sh───spk_wrap_frame───spawn─┬─pycuerun
│ └─258*[{spawn}]
├─2*[time───su───tcsh───sh───spk_wrap_frame───spawn─┬─pycuerun───python]
│ └─18*[{spawn}]]
├─time───su───tcsh───sh───spk_wrap_frame───spawn─┬─pycuerun───python───frame_script───15*[{frame_script}]
│ └─18*[{spawn}]
├─time───su───tcsh───sh───spk_wrap_frame───spawn─┬─pycuerun───python───frame_script
│ └─18*[{spawn}]
├─time
└─26*[{python}]
```

As we can see, a lot of frame_script processes are jammed. After analysis (I spare you the details), they are trying to write to stdout which is connected through a pipe to the parent process. The parent process seems to be not reading enough data from the pipe (or jammed somehow) and the spotless process block on the write. These processes will never return. (The reason for this particular case seems to be a lack of space on /tmp devices which cascaded into this problem).

**PROPOSED SOLUTION**

The solution is to change the "subProcess.wait" in RQD to something more elaborate. Note that the procedure that "communicates" with stderr/stdout just before the process wait has to be done in an ASYNC function.
Here is an example implementation algorithm:

```
import os
import signal
import psutil
from subprocess import TimeoutExpired

def wait_with_idle_kill(proc, cpu_threshold=0.01, interval=3, max_idle_checks=3):
"""
Waits for the process to finish. If total normalized CPU usage (proc + children)
is below `cpu_threshold` (0.0 to 1.0) for `max_idle_checks` consecutive checks,
kills the process group.

The first wait is quick to avoid unnecessary delay. It is the minimum between 10 seconds
and the user-defined `interval`, to give the process a chance to initialize quickly
before starting to monitor its CPU usage.

The first CPU check is skipped to avoid an uninitialized value.
"""
pid = proc.pid
idle_count = 0
num_cores = psutil.cpu_count(logical=True)
first_check = True
first_wait = min(interval, 10) # First wait interval (quick check)

# Ensure it's in its own process group
try:
os.setpgid(pid, pid)
except Exception:
pass

while True:
try:
# First wait is quick (min of 10 seconds or user-defined interval)
proc.wait(timeout=first_wait if first_check else interval)
print(f"Process exited with code {proc.returncode}")
break
except TimeoutExpired:
try:
ps_proc = psutil.Process(pid)
all_procs = [ps_proc] + ps_proc.children(recursive=True)

# Non-blocking CPU usage read. The cpu_percent will
total_cpu_percent = sum(
p.cpu_percent(interval=None)
for p in all_procs
if p.is_running()
)
normalized_cpu = total_cpu_percent / (100.0 * num_cores)

if first_check:
# Skip the first CPU check to avoid uninitialized value (0.0)
print("Skipping first CPU check (initialization only)")
first_check = False
continue

print(f"Normalized CPU usage: {normalized_cpu:.3f}")

if normalized_cpu < cpu_threshold:
idle_count += 1
print(f"Idle count: {idle_count}/{max_idle_checks}")
else:
idle_count = 0

if idle_count >= max_idle_checks:
print("Idle threshold exceeded. Killing process group.")
os.killpg(os.getpgid(pid), signal.SIGKILL)
break

except psutil.NoSuchProcess:
print("Process already exited.")
break
```

Start by locating RQD's subprocess.wait call and the preceding stdout/stderr communication path; the issue says that communication must run in an async function. Review the two failure modes and the proposed wait_with_idle_kill algorithm, then verify that crashed or de-parented processes are detected and that blocked output pipes no longer leave machines wedged.

Written by the indexing model from the issue text.