Intermittent MPI test failures

@fschlimb is already working on this.

Since Aug 10, 2026.

The `examples/mpi/feed-forward-mpi.py` randomly fails.
Root cause should be investigated and the test made stable before reenabling it.

Example failed job dump:
```
-- Testing: 28 tests, 4 workers --
PASS: Lighthouse test suite :: examples/basic/transform_a_payload_according_to_a_schedule.py (1 of 28)
PASS: Lighthouse test suite :: examples/ingress/mlir_gen/generate-linalg-3layer-mlp.sh (2 of 28)
PASS: Lighthouse test suite :: examples/basic/compile_and_run.py (3 of 28)
PASS: Lighthouse test suite :: examples/ingress/torch/mlp_from_file.py (4 of 28)
PASS: Lighthouse test suite :: examples/ingress/torch/mlp_from_model.py (5 of 28)
PASS: Lighthouse test suite :: examples/ingress/torch/compile_model.py (6 of 28)
FAIL: Lighthouse test suite :: examples/mpi/feed-forward-mpi.py (7 of 28)
******************** TEST 'Lighthouse test suite :: examples/mpi/feed-forward-mpi.py' FAILED ********************
Exit Code: 1

Command Output (stderr):
--
RUN: at line 2: mpirun -n 4 python /home/runner/work/lighthouse/lighthouse/examples/mpi/feed-forward-mpi.py --mpilib=/home/runner/work/lighthouse/lighthouse/.venv/lib/libmpi.so.12 | /usr/bin/FileCheck-18 /home/runner/work/lighthouse/lighthouse/examples/mpi/feed-forward-mpi.py
+ mpirun -n 4 python /home/runner/work/lighthouse/lighthouse/examples/mpi/feed-forward-mpi.py --mpilib=/home/runner/work/lighthouse/lighthouse/.venv/lib/libmpi.so.12
+ /usr/bin/FileCheck-18 /home/runner/work/lighthouse/lighthouse/examples/mpi/feed-forward-mpi.py
Abort(404899215): Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71).: MPI_Init_thread(argc=(nil), argv=(nil), required=3, provided=0x7ffe8a676060) failed
MPII_Init_thread(261)....:
MPID_Init(549)...........:
MPIDI_UCX_init_local(234):
MPIDI_UCX_init_worker(86): ucx function returned with failed status(ucx_init.c 86 MPIDI_UCX_init_worker Input/output error)
Abort(1008878991): Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71).: MPI_Init_thread(argc=(nil), argv=(nil), required=3, provided=0x7ffc2b7a8fc0) failed
MPII_Init_thread(261)....:
MPID_Init(549)...........:
MPIDI_UCX_init_local(234):
MPIDI_UCX_init_worker(86): ucx function returned with failed status(ucx_init.c 86 MPIDI_UCX_init_worker Input/output error)
Abort(136463759): Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(71).: MPI_Init_thread(argc=(nil), argv=(nil), required=3, provided=0x7ffe591ac650) failed
MPII_Init_thread(261)....:
MPID_Init(549)...........:
MPIDI_UCX_init_local(234):
MPIDI_UCX_init_worker(86): ucx function returned with failed status(ucx_init.c 86 MPIDI_UCX_init_worker Input/output error)
/home/runner/work/lighthouse/lighthouse/examples/mpi/feed-forward-mpi.py:5:10: error: CHECK: expected string not found in input
# CHECK: PASSED
^
:1:1: note: scanning from here
[1785831715.877809] [runnervmvrwv9:2894 :0] ib_iface.c:1316 UCX ERROR mana_0: iface 0x3cb598a0 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
^
:9:49: note: possible intended match here
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
^

Input file:
Check file: /home/runner/work/lighthouse/lighthouse/examples/mpi/feed-forward-mpi.py

-dump-input=help explains the following input dump.

Input was:
<<<<<<
1: [1785831715.877809] [runnervmvrwv9:2894 :0] ib_iface.c:1316 UCX ERROR mana_0: iface 0x3cb598a0 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
check:5'0 X~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ error: no match found
2: [1785831715.878738] [runnervmvrwv9:2894 :0] ucp_worker.c:1415 UCX ERROR uct_iface_open(ud_verbs/mana_0:1) failed: Input/output error
check:5'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3: [1785831715.879088] [runnervmvrwv9:2891 :0] ib_iface.c:1316 UCX ERROR mana_0: iface 0x384043c0 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
check:5'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4: [1785831715.880186] [runnervmvrwv9:2892 :0] ib_iface.c:1316 UCX ERROR mana_0: iface 0x1f45cc60 failed to create UD QP TX wr:256 sge:6 inl:64 resp:0 RX wr:4096 sge:1 resp:0 failed: Operation not supported
check:5'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5: [1785831715.880779] [runnervmvrwv9:2891 :0] ucp_worker.c:1415 UCX ERROR uct_iface_open(ud_verbs/mana_0:1) failed: Input/output error
check:5'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
6: [1785831715.881213] [runnervmvrwv9:2892 :0] ucp_worker.c:1415 UCX ERROR uct_iface_open(ud_verbs/mana_0:1) failed: Input/output error
check:5'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
7:
check:5'0 ~
8: ===================================================================================
check:5'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
9: = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
check:5'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
check:5'1 ? possible intended match
10: = PID 2893 RUNNING AT runnervmvrwv9
check:5'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
11: = EXIT CODE: 9
check:5'0 ~~~~~~~~~~~~~~~
12: = CLEANING UP REMAINING PROCESSES
check:5'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
13: = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
check:5'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
14: ===================================================================================
check:5'0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.
.
.
>>>>>>

--

********************
PASS: Lighthouse test suite :: examples/llama/test_llama3.py (8 of 28)
PASS: Lighthouse test suite :: examples/xegpu/fused_attention.py (9 of 28)
PASS: Lighthouse test suite :: examples/xegpu/layer_norm.py (10 of 28)
PASS: Lighthouse test suite :: examples/llama/torch_ingress.py (11 of 28)
PASS: Lighthouse test suite :: examples/xegpu/matmul.py (12 of 28)
PASS: Lighthouse test suite :: examples/xegpu/nanoGPT.py (13 of 28)
PASS: Lighthouse test suite :: examples/xegpu/softmax.py (14 of 28)
UNSUPPORTED: Lighthouse test suite :: examples/xegpu/torch_matmul.py (15 of 28)
PASS: Lighthouse test suite :: examples/xegpu/tune_matmul_costmodel.py (16 of 28)
PASS: Lighthouse test suite :: test/dialects/dialect_loading.py (17 of 28)
PASS: Lighthouse test suite :: test/dialects/transform_ext.py (18 of 28)
PASS: Lighthouse test suite :: test/dialects/transform_ext_subview.py (19 of 28)
PASS: Lighthouse test suite :: examples/xegpu/mlp.py (20 of 28)
PASS: Lighthouse test suite :: test/pipeline/find_pipeline.py (21 of 28)
PASS: Lighthouse test suite :: test/run/pipeline-check.mlir (22 of 28)
PASS: Lighthouse test suite :: test/transform/test_foreach.py (23 of 28)
PASS: Lighthouse test suite :: test/transform/test_pack_lowering.py (24 of 28)
PASS: Lighthouse test suite :: test/transform/test_tile_size_ops.py (25 of 28)
PASS: Lighthouse test suite :: test/opt/pipeline-check.mlir (26 of 28)
PASS: Lighthouse test suite :: examples/xegpu/kernel_bench.py (27 of 28)
PASS: Lighthouse test suite :: examples/KernelBench/test-kernel-bench.py (28 of 28)
********************
Failed Tests (1):
Lighthouse test suite :: examples/mpi/feed-forward-mpi.py
```