The runner doesn't finish when a task fails

Describe the bug
I one of our systemd repositories, I came across a strange issue where if a step fails, the runner correctly reports the step failed, but then it remains stuck in the "running" state for several hours until it's killed by the global watchdog. Also, killing the job manually in this state is also... strange, since hitting "Cancel workflow" once doesn't work, one has to hit it many times to make it kill the job.

This feels like it's the same issue as https://github.com/actions/runner/issues/700 (and given the last comment from @rubencodes it might still be unresolved).

To Reproduce
Steps to reproduce the behavior:
I still can't pinpoint what exactly causes this issue, but I can reproduce it reliably in this PR: https://github.com/redhat-plumbers/systemd-rhel8/pull/302, where the problematic steps looks like sudo -E script.sh -> docker exec ... ninja -C build test.

Expected behavior
The job should report a failure immediately when a step fails (or in a reasonable time frame).

Runner Version and Platform

2.294.0

OS of the machine running the runner? OSX/Windows/Linux/...
Ubuntu 22.04

What's not working?

The job reports it failed, e.g.:

Summary of Failures:
375/377 fuzz-varlink_oss-fuzz-14688_address                                        FAIL             0.01s   exit status 127
376/377 fuzz-varlink_oss-fuzz-14708_address                                        FAIL             0.01s   exit status 127
Ok:                 365 
Expected Fail:      0   
Fail:               2   
Unexpected Pass:    0   
Skipped:            10  
Timeout:            0   
Full log written to /build/build/meson-logs/testlog.txt
FAILED: meson-test 
/usr/bin/meson test --no-rebuild --print-errorlogs
ninja: build stopped: subcommand failed.
Error: Process completed with exit code 1.

but then remains stuck for several hours in "running" state until it's eventually cancelled by the global watchdog:

The job itself: https://github.com/redhat-plumbers/systemd-rhel8/actions/runs/2690828471/attempts/2

A bit of further debugging shows that the information about the failed process correctly "bubbles up" the tree all the way to the runner, which then gets stuck:

 Summary of Failures:
154/372 test-bpf                                                                   FAIL             0.03s   killed by signal 6 SIGABRT
Ok:                 362 
Expected Fail:      0   
Fail:               1   
Unexpected Pass:    0   
Skipped:            9   
Timeout:            0   
Full log written to /build/build/meson-logs/testlog.txt
FAILED: meson-test 
/usr/bin/meson test --no-rebuild --print-errorlogs
ninja: build stopped: subcommand failed.
+ at_exit
+ echo 'Hello from at_exit()'
+ pstree -Aapust 3388
Hello from at_exit()
systemd,1
  `-provisioner,668 --agentdirectory /home/runner/runners --settings /opt/runner/provisioner/.settings
      `-Runner.Listener,2035,runner run
          `-Runner.Worker,2057 spawnclient 112 115
              `-bash,3386 -e /home/runner/work/_temp/73c812fa-ef6d-4cb8-a92a-db5c79d10865.sh
                  `-sudo,3387,root -E .github/workflows/unit_tests.sh RUN_GCC
                      `-unit_tests.sh,3388 .github/workflows/unit_tests.sh RUN_GCC
                          `-pstree,13946 -Aapust 3388
+ exit 1
Error: Process completed with exit code 1.
<here the runner stops responding>

Start by tracing the Runner.Listener and Runner.Worker execution path described in the report, then reproduce the failure through the linked systemd-rhel8 pull request and workflow run. Done means a failed step promptly marks the job failed and workflow cancellation works without repeated requests.