bst2 hangs after crashing

Trying to port carbonOS to bst2 (specifically, 2.0.1). Using `project.refs`. I've run into this in (at least):
- a `fedora:38` distrobox w/ buildstrem `pip install`'d into /usr/local
- a `docker.io/buildstream/buildstream:latest` distrobox
- a `docker.io/buildstream/buildstream:dev` toolbox

After I run into #1851 and #1852, I try to Ctrl+C out of buildstream and am sometimes met with the following. I get one of these for every ongoing track, except for the one that failed in the above issues. Afterwards, `bst` hangs forever and ignores Ctrl+C. I need to `kill -9 bst` from another terminal.

```
Track failure on element: pkgs/librsvg.bst

Choose one of the following options:
(c)ontinue - Continue queueing jobs as much as possible
(q)uit - Exit after all ongoing jobs complete
(t)erminate - Terminate any ongoing jobs and exit
(r)etry - Retry this job
(l)og - View the full log file

Pressing ^C will terminate jobs and exit
Choice: [continue]: t

Terminating all jobs

[--:--:--][????????][ track:pkgs/gettext.bst ] INFO Found tarball: gettext/gettext-0.22.tar.xz (hash: 0e60393a47061567b46875b249b7d2788b092d6457d656145bb0e7e6a3e26d93)
[00:00:17][????????][ track:pkgs/gettext.bst ] SUCCESS carbonOS/pkgs-gettext/????????-track.2533421.log
[--:--:--][????????][ track:pkgs/gcc.bst ] INFO Found tarball: gcc/gcc-13.2.0/gcc-13.2.0.tar.xz (hash: e275e76442a6067341a27f04c5c6b83d8613144004c0413528863dc6b5c743da)
[00:00:29][????????][ track:pkgs/gcc.bst ] SUCCESS carbonOS/pkgs-gcc/????????-track.2533421.log
[--:--:--][????????][ main:pkgs/gcc.bst ] STATUS Track terminating
[--:--:--][????????][ main:pkgs/gettext.bst ] STATUS Track terminating
[--:--:--][????????][ main:pkgs/libpinyin.bst ] STATUS Track terminating
[--:--:--][????????][ main:pkgs/libpulse.bst ] STATUS Track terminating
[--:--:--][????????][ main:pkgs/libseccomp.bst ] STATUS Track terminating
[--:--:--][????????][ main:pkgs/libsecret.bst ] STATUS Track terminating
[--:--:--][????????][ main:pkgs/libsigcpp.bst ] STATUS Track terminating
[--:--:--][????????][ main:pkgs/libslirp.bst ] STATUS Track terminating
[--:--:--][????????][ main:pkgs/libsmbios.bst ] STATUS Track terminating
[00:00:00][????????][ track:pkgs/libslirp.bst ] BUG Track

An unhandled exception occured:

Traceback (most recent call last):
File "/usr/local/lib64/python3.11/site-packages/psutil/_pslinux.py", line 1653, in wrapper
return fun(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/psutil/_common.py", line 480, in wrapper
raise raise_from(err, None)
^^^^^^^^^^^^^^^^^^^^^
File "", line 3, in raise_from
File "/usr/local/lib64/python3.11/site-packages/psutil/_common.py", line 478, in wrapper
return fun(self)
^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/psutil/_pslinux.py", line 1695, in _parse_stat_file
data = bcat("%s/%s/stat" % (self._procfs_path, self.pid))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/psutil/_common.py", line 813, in bcat
return cat(fname, fallback=fallback, _open=open_binary)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/psutil/_common.py", line 801, in cat
with _open(fname) as f:
^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/psutil/_common.py", line 765, in open_binary
return open(fname, "rb", buffering=FILE_READ_BUFFER_SIZE)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/proc/2572392/stat'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib64/python3.11/site-packages/psutil/__init__.py", line 361, in _init
self.create_time()
File "/usr/local/lib64/python3.11/site-packages/psutil/__init__.py", line 719, in create_time
self._create_time = self._proc.create_time()
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/psutil/_pslinux.py", line 1653, in wrapper
return fun(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/psutil/_pslinux.py", line 1863, in create_time
ctime = float(self._parse_stat_file()['create_time'])
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/psutil/_pslinux.py", line 1660, in wrapper
raise NoSuchProcess(self.pid, self._name)
psutil.NoSuchProcess: process no longer exists (pid=2572392)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib64/python3.11/site-packages/buildstream/_scheduler/jobs/job.py", line 438, in child_action
result = self.child_process() # pylint: disable=assignment-from-no-return
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/buildstream/_scheduler/jobs/elementjob.py", line 91, in child_process
return self._action_cb(self._element)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/buildstream/_scheduler/queues/trackqueue.py", line 65, in _track_element
return element._track()
^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/buildstream/element.py", line 1390, in _track
return self.__sources.track(self._get_workspace())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/buildstream/_elementsources.py", line 97, in track
new_ref = source._track()
^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/buildstream/source.py", line 1202, in _track
new_ref = self.__do_track()
^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/buildstream/source.py", line 1430, in __do_track
return self.track(**kwargs)
^^^^^^^^^^^^^^^^^^^^
File "/var/home/adrian/Projects/build-meta/.bst/staged-junctions/junctions/bst-plugins-experimental.bst/7a4409718df665fb7e5ed800186b38fc904362557ad3c25d5be69de503cce030/src/bst_plugins_experimental/sources/git_tag.py", line 940, in track
self.mirror._fetch()
File "/var/home/adrian/Projects/build-meta/.bst/staged-junctions/junctions/bst-plugins-experimental.bst/7a4409718df665fb7e5ed800186b38fc904362557ad3c25d5be69de503cce030/src/bst_plugins_experimental/sources/git_tag.py", line 394, in _fetch
self.source.call(
File "/usr/local/lib64/python3.11/site-packages/buildstream/plugin.py", line 728, in call
exit_code, _ = self.__call(
^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/buildstream/plugin.py", line 950, in __call
exit_code, output = utils._call(args, cwd=cwd, env=env, stdin=stdin, stdout=stdout, stderr=stderr)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/buildstream/utils.py", line 1435, in _call
with _signals.suspendable(suspend_proc, resume_proc), _signals.terminator(kill_proc), subprocess.Popen(
File "/usr/lib64/python3.11/contextlib.py", line 155, in __exit__
self.gen.throw(typ, value, traceback)
File "/usr/local/lib64/python3.11/site-packages/buildstream/_signals.py", line 113, in terminator
terminate_func()
File "/usr/local/lib64/python3.11/site-packages/buildstream/utils.py", line 1423, in kill_proc
_kill_process_tree(process.pid)
File "/usr/local/lib64/python3.11/site-packages/buildstream/utils.py", line 1342, in _kill_process_tree
proc = psutil.Process(pid)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/psutil/__init__.py", line 332, in __init__
self._init(pid)
File "/usr/local/lib64/python3.11/site-packages/psutil/__init__.py", line 373, in _init
raise NoSuchProcess(pid, msg='process PID not found')
psutil.NoSuchProcess: process PID not found (pid=2572392)

[00:00:01][????????][ track:pkgs/libpulse.bst ] BUG Track
...
[00:00:00][????????][ track:pkgs/libsigcpp.bst ] BUG Track
...
[--:--:--][????????][ main:pkgs/libsmbios.bst ] INFO Job terminated
[--:--:--][????????][ main:pkgs/libseccomp.bst ] INFO Job terminated
[--:--:--][????????][ main:pkgs/libpinyin.bst ] INFO Job terminated
[00:00:00][????????][ track:pkgs/libsecret.bst ] BUG Track
...
[00:00:23][ ][ main:core activity ] FAILURE Track
```

Similar error from a different invocation of bst, but with a slightly different stack trace:

```
[00:00:21][????????][ track:pkgs/apps/nautilus.bst ] BUG Track

An unhandled exception occured:

Traceback (most recent call last):
File "/usr/local/lib64/python3.11/site-packages/psutil/_pslinux.py", line 1653, in wrapper
return fun(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/psutil/_common.py", line 480, in wrapper
raise raise_from(err, None)
^^^^^^^^^^^^^^^^^^^^^
File "", line 3, in raise_from
File "/usr/local/lib64/python3.11/site-packages/psutil/_common.py", line 478, in wrapper
return fun(self)
^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/psutil/_pslinux.py", line 1695, in _parse_stat_file
data = bcat("%s/%s/stat" % (self._procfs_path, self.pid))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/psutil/_common.py", line 813, in bcat
return cat(fname, fallback=fallback, _open=open_binary)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/psutil/_common.py", line 801, in cat
with _open(fname) as f:
^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/psutil/_common.py", line 765, in open_binary
return open(fname, "rb", buffering=FILE_READ_BUFFER_SIZE)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/proc/2463427/stat'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib64/python3.11/site-packages/psutil/__init__.py", line 361, in _init
self.create_time()
File "/usr/local/lib64/python3.11/site-packages/psutil/__init__.py", line 719, in create_time
self._create_time = self._proc.create_time()
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/psutil/_pslinux.py", line 1653, in wrapper
return fun(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/psutil/_pslinux.py", line 1863, in create_time
ctime = float(self._parse_stat_file()['create_time'])
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/psutil/_pslinux.py", line 1660, in wrapper
raise NoSuchProcess(self.pid, self._name)
psutil.NoSuchProcess: process no longer exists (pid=2463427)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib64/python3.11/site-packages/buildstream/_scheduler/jobs/job.py", line 438, in child_action
result = self.child_process() # pylint: disable=assignment-from-no-return
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/buildstream/_scheduler/jobs/elementjob.py", line 91, in child_process
return self._action_cb(self._element)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/buildstream/_scheduler/queues/trackqueue.py", line 65, in _track_element
return element._track()
^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/buildstream/element.py", line 1390, in _track
return self.__sources.track(self._get_workspace())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/buildstream/_elementsources.py", line 97, in track
new_ref = source._track()
^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/buildstream/source.py", line 1202, in _track
new_ref = self.__do_track()
^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/buildstream/source.py", line 1430, in __do_track
return self.track(**kwargs)
^^^^^^^^^^^^^^^^^^^^
File "/var/home/adrian/Projects/build-meta/.bst/staged-junctions/junctions/bst-plugins-experimental.bst/7a4409718df665fb7e5ed800186b38fc904362557ad3c25d5be69de503cce030/src/bst_plugins_experimental/sources/git_tag.py", line 939, in track
self.mirror.ensure_trackable()
File "/var/home/adrian/Projects/build-meta/.bst/staged-junctions/junctions/bst-plugins-experimental.bst/7a4409718df665fb7e5ed800186b38fc904362557ad3c25d5be69de503cce030/src/bst_plugins_experimental/sources/git_tag.py", line 334, in ensure_trackable
self.source.call(
File "/usr/local/lib64/python3.11/site-packages/buildstream/plugin.py", line 728, in call
exit_code, _ = self.__call(
^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/buildstream/plugin.py", line 950, in __call
exit_code, output = utils._call(args, cwd=cwd, env=env, stdin=stdin, stdout=stdout, stderr=stderr)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/buildstream/utils.py", line 1435, in _call
with _signals.suspendable(suspend_proc, resume_proc), _signals.terminator(kill_proc), subprocess.Popen(
File "/usr/lib64/python3.11/contextlib.py", line 155, in __exit__
self.gen.throw(typ, value, traceback)
File "/usr/local/lib64/python3.11/site-packages/buildstream/_signals.py", line 113, in terminator
terminate_func()
File "/usr/local/lib64/python3.11/site-packages/buildstream/utils.py", line 1423, in kill_proc
_kill_process_tree(process.pid)
File "/usr/local/lib64/python3.11/site-packages/buildstream/utils.py", line 1342, in _kill_process_tree
proc = psutil.Process(pid)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/psutil/__init__.py", line 332, in __init__
self._init(pid)
File "/usr/local/lib64/python3.11/site-packages/psutil/__init__.py", line 373, in _init
raise NoSuchProcess(pid, msg='process PID not found')
psutil.NoSuchProcess: process PID not found (pid=2463427)
```

Start by reproducing cancellation after a track failure, then read buildstream/utils.py, buildstream/_signals.py, and the scheduler job paths shown in the traceback. Check how process termination handles an already-exited PID and verify that Ctrl+C exits without an unhandled exception or a hung bst process.

Written by the indexing model from the issue text.