JuliaParallel / JuliaParallel/Elemental.jl
LAV Solver Stalls with Custom LPAffineCtrl Configuration on Multi-Rank MPI
- Dominant language
- Julia
- Stars
- 85
- Forks
- 13
- PR merge metrics
- No merged PRs in 30d
Description
### Description
When running the distributed **LAV (Least Absolute Value)** solver test with a custom `LPAffineCtrl` configuration where `print`, `progress`, and `time` flags are enabled, the test stalls and exceeds the 5-minute watchdog timeout. This occurs specifically when executed across multiple MPI ranks (e.g., using `mpiexec` with 4 processes).
---
### Environment
* **Julia Versions:** 1.10 and release candidate builds
* **MPI Setup:** `MPICH_jll` or `OpenMPI` via `mpiexec -n 4`
* **Trigger:** Fresh dependency resolution (common in CI/CD environments)
---
### Problem Details
#### Custom Control Configuration Causes Stall
A custom `LPAffineCtrl` is constructed using `Float64` with an embedded `MehrotraCtrl`. The following configuration triggers the hang:
* `print`, `progress`, and `time` flags enabled (both in `MehrotraCtrl` and top-level).
* `outerEquil` set to `true`.
* Passing this object into `El.lav` during a distributed run.
#### Workaround: Default Behavior Works
Calling `El.lav(A, b)` without passing a custom control object completes successfully within the timeout and does not hang.
---
### Root Cause Hypothesis
**I/O Contention & Synchronization:** Multi-rank logging and progress instrumentation likely introduce bottlenecks. When multiple MPI processes attempt to emit output simultaneously, it may lead to deadlocks or extreme performance degradation.
**Supporting Observations:**
* **Default Success:** The test completes immediately without logging/progress flags.
* **MPI Specificity:** The stall is exclusive to multi-rank settings with enabled instrumentation.
* **Environment Factor:** The issue is more prevalent after fresh dependency resolution; cached local builds are less susceptible.
---
### Impact
To prevent CI timeouts, the test suite currently uses a simplified solver invocation. This reduces test coverage for:
1. **Custom solver control options** (`LPAffineCtrl`).
2. **Progress reporting** functionality.
3. **Timing and performance instrumentation** in distributed environments.
---
### Suggested Investigation
* **Profiling:** Profile solver behavior in multi-rank mode with logging enabled vs. disabled.
* **I/O Inspection:** Inspect MPI synchronization and I/O patterns inside `El.lav` when flags are active.
* **Algorithm Isolation:** Determine if this is specific to the **Mehrotra interior point algorithm** or affects other configurations.
* **Scaling Tests:** Test across different MPI process counts (2, 8, 16) to observe scaling behavior.
---
### References
* **Fix History:** Previous stabilization involved reducing the test matrix size (50 to 12) and reverting to the default solver path.
* **Test File:** `lav.jl` (lines 5–6 problem size, line 97 solver invocation).
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.