Hydrofracture model parallel simulation results mismatch using different number of processes

Build: GEOS (version: 1.1.0 (develop, sha1: 3c4000e3c)) and TPL (with this pull request: https://github.com/GEOS-DEV/thirdPartyLibs/pull/304/commits/1f91b9f60d96b58fd0f4cd419c98fe11e15f8df4, with hypre version: v2.32.0-34-g72b59c958 (fix-csr-filter)) compiled with gcc 13.2.0 and either Intel MPI 2021.14 or Open MPI v4.1.7rc1 (repo rev: v4.1.5-164-g4292ae09f9).

For the attached hydrofracture model (simultaneous growth of three hydraulic fractures), results from runs with different number of processes on different number of computing nodes (each node has 120 cores) with either Intel MPI or Open MPI mismatched, with linear tolerance=1.e-8 and the nonlinear tolerance=1.e-6. Below is a summary of the different runs (the single process run is still running after 8 days, no result as of now):

a)For Intel MPI, 16-, 32-, and 64-proc simulation results do not match each other. But 16-proc runs with 1 or 4 nodes match each other, 64-proc runs with 1 or 4 nodes match each other, 32-proc runs with 1 or 4 nodes do not match each other. For the 32-proc run with 4 nodes, the last time step (from time=1200.0 s to 1200.1 s) failed to convergence because linear solver hits max number of iterations and does not reduce residual.
b)For Open MPI, 32-proc run on 4 nodes failed, 32-proc run on 1 node completed, but result differs from 32-proc run with Intel MPI on 1 or 4 nodes.

The failed 32-proc run on 4 nodes with Open MPI has the following error message:

Attempt:  0, ConfigurationIter:  0, NewtonIter:  4
    ( R ) = ( 1.45e-02 )
    MGR preconditioner: numComponentsPerField = [3, 1]
    Linear Solver | Success | Unknowns: 203,838 | Nonzeros: -9,204,516,431,592,302,017 | Iterations: 142 | Final Rel Res: 1.1256e+05 | Setup Time: 0.127 s | Solve Time: 0.919 s
    Last LinSolve(iter,res) = ( 142, 1.13e+05 )
    hydrofracture: Global solution scaling factor = -201.0974375851566

[1742917246.180286] [ccnpuscu00000y:7591 :0] tl_ucp_coll.c:146 TL_UCP ERROR failure in recv completion Message truncated
[1742917246.180315] [ccnpuscu00000y:7591 :0] tl_ucp_coll.c:146 TL_UCP ERROR failure in recv completion Message truncated
[1742917246.180737] [ccnpuscu000002:7298 :0] tl_ucp_coll.c:146 TL_UCP ERROR failure in recv completion Message truncated
[1742917246.180787] [ccnpuscu000002:7298 :0] tl_ucp_coll.c:146 TL_UCP ERROR failure in recv completion Message truncated

Could you please look into what could be causing the issue? Thanks.

synthetic_3fracs_coarse_mesh.tar.gz