job_dispatch should try to catch leftover grandchildren processes

In
https://github.com/equinor/ert/blob/76ee3fe27de07086c96007fb1c8089408af8ef39/src/_ert_job_runner/job.py#L24

there is a loop over each forward model.

If a forward model (called a "job" in this source code) launches a subprocess, job_dispatch can be fooled into thinking all forward models are finished.

When this situation arises on an LSF cluster, the Ert GUI will show all forward models as finished (green) but there will be a yellow border around the realization in the detailed view. This indicates that LSF claims the realization is still running (which is correct) as the grandchildren process can still be found on the compute node, with a parent pid set to 1.

Since LSF will never report the job as finished (assuming it is hanging), Ert will wait until potentially MAX_RUNNING is hit and cleanup will occur.

Start at src/_ert_job_runner/job.py around line 24, where job_dispatch loops over forward models. Investigate how subprocess descendants are handled when a job appears finished, especially the orphaned-grandchild case described for LSF. Done means dispatch does not report the realization as complete while a leftover descendant is still running, avoiding the GUI and cleanup inconsistency.

Written by the indexing model from the issue text.