`hyrax` doesn't terminate train step on failure while on gpu

Nobody has claimed this yet.

**Bug report**
While running some training on gpu (ampere) on usdf, I've run into a minor problem where the training run fails due to an exception, but does not terminate the full python process. This is with running hyrax as a script through the cli.

my script file
```python
from hyrax import Hyrax
import numpy as np

if __name__ == "__main__":
hyrax_instance = Hyrax(config_file="/sdf/home/m/maxwest/rubin-user/kbmod-ml/user_config.toml")

hyrax_instance.config["data_set"]["seed"] = 101010101

# Set the number of epochs to run over so we can test more easily
hyrax_instance.config["train"]["epochs"] = 4
hyrax_instance.config["torch.optim.SGD"]["lr"] = 0.005
hyrax_instance.config["torch.optim.SGD"]["momentum"] = 0.05

hyrax_instance.config["train"]["weights_filepath"] = "cnn_weights.pth"
hyrax_instance.train()

```

it seems like the error is captured by the logger and something is spun down, but the process remains running until I `ctrl+c`.

errors (logger is dumped because I added a manual print):
```
[2025-05-13 10:52:37,392 hyrax.pytorch_ignite:INFO] Training model on device: cuda
tensor([[ 0.5175, -0.1294],
[ 0.1998, -0.0746],
[ 0.2033, 0.0429],
[ 0.1845, 0.0097],
[ 0.1937, -0.0139],
[ 0.3842, -0.4312]], device='cuda:0', grad_fn=)
tensor([0, 1, 1, 1, 1, 0], device='cuda:0', dtype=torch.int8)
Current run is terminating due to exception: "host_softmax" not implemented for 'Char'
Engine run is terminating due to exception: "host_softmax" not implemented for 'Char'
2025/05/13 10:52:37 INFO mlflow.system_metrics.system_metrics_monitor: Stopping system metrics monitoring...
2025/05/13 10:52:37 INFO mlflow.system_metrics.system_metrics_monitor: Successfully terminated system metrics monitoring!
Traceback (most recent call last):
File "/sdf/data/rubin/user/maxwest/kbml_runs/run_train.py", line 15, in
hyrax_instance.train()
File "/sdf/data/rubin/user/maxwest/hyrax/src/hyrax/hyrax.py", line 161, in train
return run(config=self.config, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/sdf/data/rubin/user/maxwest/hyrax/src/hyrax/train.py", line 72, in run
trainer.run(train_data_loader, max_epochs=config["train"]["epochs"])
File "/sdf/data/rubin/user/maxwest/kbml_env/lib/python3.11/site-packages/ignite/engine/engine.py", line 905, in run
return self._internal_run()
^^^^^^^^^^^^^^^^^^^^
File "/sdf/data/rubin/user/maxwest/kbml_env/lib/python3.11/site-packages/ignite/engine/engine.py", line 948, in _internal_run
return next(self._internal_run_generator)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/sdf/data/rubin/user/maxwest/kbml_env/lib/python3.11/site-packages/ignite/engine/engine.py", line 1023, in _internal_run_as_gen
self._handle_exception(e)
File "/sdf/data/rubin/user/maxwest/kbml_env/lib/python3.11/site-packages/ignite/engine/engine.py", line 660, in _handle_exception
raise e
File "/sdf/data/rubin/user/maxwest/kbml_env/lib/python3.11/site-packages/ignite/engine/engine.py", line 972, in _internal_run_as_gen
epoch_time_taken += yield from self._run_once_on_dataset_as_gen()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/sdf/data/rubin/user/maxwest/kbml_env/lib/python3.11/site-packages/ignite/engine/engine.py", line 1128, in _run_once_on_dataset_as_gen
self._handle_exception(e)
File "/sdf/data/rubin/user/maxwest/kbml_env/lib/python3.11/site-packages/ignite/engine/engine.py", line 660, in _handle_exception
raise e
File "/sdf/data/rubin/user/maxwest/kbml_env/lib/python3.11/site-packages/ignite/engine/engine.py", line 1110, in _run_once_on_dataset_as_gen
self.state.output = self._process_function(self, self.state.batch)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/sdf/data/rubin/user/maxwest/hyrax/src/hyrax/pytorch_ignite.py", line 302, in _inner_loop
return func(batch)
^^^^^^^^^^^
File "/sdf/data/rubin/user/maxwest/kbmod-ml/src/kbmod_ml/models/resnet50.py", line 52, in train_step
loss = self.criterion(outputs.type(torch.int8), labels)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/sdf/data/rubin/user/maxwest/kbml_env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/sdf/data/rubin/user/maxwest/kbml_env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/sdf/data/rubin/user/maxwest/kbml_env/lib/python3.11/site-packages/torch/nn/modules/loss.py", line 1297, in forward
return F.cross_entropy(
^^^^^^^^^^^^^^^^
File "/sdf/data/rubin/user/maxwest/kbml_env/lib/python3.11/site-packages/torch/nn/functional.py", line 3494, in cross_entropy
return torch._C._nn.cross_entropy_loss(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: "host_softmax" not implemented for 'Char'
^CException ignored in:
Traceback (most recent call last):
File "/sdf/data/rubin/user/maxwest/lsst_stack/conda/envs/lsst-scipipe-9.0.0/lib/python3.11/threading.py", line 1590, in _shutdown
lock.acquire()
KeyboardInterrupt:

```
as you can see, the process fails and stops doing anything but doesn't quit until I manually `ctrl+c`.

**Before submitting**
Please check the following:

- [x] I have described the situation in which the bug arose, including what code was executed, information about my environment, and any applicable data others will need to reproduce the problem.
- [x] I have included available evidence of the unexpected behavior (including error messages, screenshots, and/or plots) as well as a description of what I expected instead.
- [x] If I have a solution in mind, I have provided an explanation and/or pseudocode and/or task list.

Reproduce the failure through the CLI script and trace the exception from hyrax/train.py into hyrax/pytorch_ignite.py, where the training engine reports termination. Check how the run returns after the exception and verify that the Python process exits without requiring Ctrl+C. Done means the logged exception and traceback remain available while the failed training process terminates.

Written by the indexing model from the issue text.