FUNCTIONALITY_TEST_SCOPFLOW_HIOP_TESTSUITE not converging on Frontier

Issue type

Bug

Issue applies to:

• PFLOW
• OPFLOW
• TCOPFLOW
• SCOPFLOW
• SOPFLOW
• CMake and build system
• Spack
• Visualization
• Documentation
• Other

Summary

Test FUNCTIONALITY_TEST_SCOPFLOW_HIOP_TESTSUITE using HiOp PriDec does not converge in finite time on Frontier. It would be good to investigate why this test is not converging.

Description how to reproduce the issue

Default log with a 300 sec timeout

With MA57, decreasing the Ipopt verbosity and increasing the PriDec verbosity, we can see how the solution bounces:

[ExaGO] Creating SCOPFlow Functionality Test
[ExaGO] Creating SCOPFLOW Functionality Test
Test Description: case9mod.m with HIOP solver/9 contingencies
[ExaGO] SCOPFLOW: Application created
[ExaGO] SCOPFLOW running with 10 subproblems (base case + 9 contingencies)
[ExaGO] SCOPFLOW: Setup completed
===============
Hiop Primal Decomposition SOLVER
===============
total number of recourse problems  9

******************************************************************************
This program contains Ipopt, a library for large-scale nonlinear optimization.
 Ipopt is released as open source code under the Eclipse Public License (EPL).
         For more information visit https://github.com/coin-or/Ipopt
******************************************************************************

real rval -3.590978747128e-01
alpd 1.000000000000e-05
solving full problem starts, iteration 0
real rval -3.590978747128e-01
previous base  4.144101531615e+03, current base 4.144101531607e+03, previous val  -3.590978747128e-01, real val -3.590978747128e-01, predicted val -3.590978747136e-01, rho_k 9.998906386702e-01
decreasing alpha ratio or decreasing minimum for quadratic coefficient
alpd 1.000000000000e-05
convergence measure 1.020370528818e-07
function val convergence measure 8.316341366866e-09
iteration           objective                   residual                   step_size                   convg
1              4.143742433732e+03            8.316341366866e-09           2.085976308132e-08         1.020370528818e-07
solving full problem starts, iteration 1
reaching error tolerance, successfully found solution
[ExaGO] -- FAIL


Test Description: case_ACTIVSg200.m with HIOP solver/20 contingencies
[ExaGO] SCOPFLOW: Application created
[ExaGO] SCOPFLOW running with 21 subproblems (base case + 20 contingencies)
[ExaGO] SCOPFLOW: Setup completed
===============
Hiop Primal Decomposition SOLVER
===============
total number of recourse problems  20
real rval -7.979952771520e+00
alpd 1.000000000000e-05
solving full problem starts, iteration 0
real rval 3.025661179515e+03
previous base  2.754959175033e+04, current base 4.180320737650e+04, previous val  -7.979952771520e+00, real val 3.025661179515e+03, predicted val -2.005871221624e+04, rho_k -2.982043978080e+00
increasing alpha ratio or increasing minimum for quadratic coefficient
This step needs to be rejected.
alpd 1.000000000000e+06
convergence measure 1.238538702132e+00
function val convergence measure 9.559764478999e+06
iteration           objective                   residual                   step_size                   convg
1              4.482886855602e+04            9.559764478999e+06           4.373913944222e+00         1.238538702132e+00
solving full problem starts, iteration 1
real rval -7.979952761702e+00
previous base  4.180320737650e+04, current base 3.469669835165e+04, previous val  3.025661179515e+03, real val -7.979952761702e+00, predicted val -1.255099322741e+06, rho_k 8.014462344376e-03
increasing alpha ratio or increasing minimum for quadratic coefficient
This step needs to be rejected.
alpd 1.000000000000e-05
convergence measure 1.517420479196e+00
function val convergence measure 2.371480804668e+06
iteration           objective                   residual                   step_size                   convg
2              3.468871839889e+04            2.371480804668e+06           1.487447015348e+00         1.517420479196e+00
solving full problem starts, iteration 2
real rval -7.979952708171e+00
previous base  3.469669835165e+04, current base 3.437695074838e+04, previous val  -7.979952761702e+00, real val -7.979952708171e+00, predicted val -6.331888539604e+06, rho_k 5.049549725366e-05
increasing alpha ratio or increasing minimum for quadratic coefficient
This step needs to be rejected.
alpd 1.000000000000e-05
convergence measure 1.000000038589e+00
function val convergence measure 6.332200307254e+06
iteration           objective                   residual                   step_size                   convg
3              3.436897079567e+04            6.332200307254e+06           2.849836328140e+00         1.000000038589e+00
solving full problem starts, iteration 3
real rval 9.149566770786e+05
previous base  3.437695074838e+04, current base 4.641765031250e+06, previous val  -7.979952708171e+00, real val 9.149566770786e+05, predicted val -8.237633312502e+13, rho_k -6.703810229114e-08
increasing alpha ratio or increasing minimum for quadratic coefficient
This step needs to be rejected.
alpd 1.000000000000e+06
convergence measure 1.527576900198e+00
function val convergence measure 8.237630883154e+13
iteration           objective                   residual                   step_size                   convg
4              5.556721708329e+06            8.237630883154e+13           6.274723993230e+00         1.527576900198e+00
solving full problem starts, iteration 4
real rval -7.980494309649e+00
previous base  4.641765031250e+06, current base 3.437693750000e+04, previous val  9.149566770786e+05, real val -7.980494309649e+00, predicted val -8.395505293986e+13, rho_k 6.577748681825e-08
increasing alpha ratio or increasing minimum for quadratic coefficient
This step needs to be rejected.
alpd 1.000000000000e-05
convergence measure 1.948976587343e+00
function val convergence measure 8.395507814829e+13
iteration           objective                   residual                   step_size                   convg
5              3.436895700569e+04            8.395507814829e+13           6.274724053782e+00         1.948976587343e+00
solving full problem starts, iteration 5
real rval 9.149566770786e+05
previous base  3.437693750000e+04, current base 4.641765015625e+06, previous val  -7.980494309649e+00, real val 9.149566770786e+05, predicted val -9.782668451756e+13, rho_k -5.645037469093e-08
increasing alpha ratio or increasing minimum for quadratic coefficient
This step needs to be rejected.
alpd 1.000000000000e+06
convergence measure 2.053752313598e+00
function val convergence measure 9.782666022408e+13
iteration           objective                   residual                   step_size                   convg
6              5.556721692704e+06            9.782666022408e+13           6.274724053782e+00         2.053752313598e+00
solving full problem starts, iteration 6
real rval -7.980494309649e+00
previous base  4.641765015625e+06, current base 3.437693750000e+04, previous val  9.149566770786e+05, real val -7.980494309649e+00, predicted val -8.395505576262e+13, rho_k 6.577748442055e-08
increasing alpha ratio or increasing minimum for quadratic coefficient
This step needs to be rejected.
alpd 1.000000000000e-05
convergence measure 1.948976578885e+00
function val convergence measure 8.395508097105e+13
iteration           objective                   residual                   step_size                   convg
7              3.436895700569e+04            8.395508097105e+13           6.274724053782e+00         1.948976578885e+00
solving full problem starts, iteration 7
real rval 9.149566770786e+05
previous base  3.437693750000e+04, current base 4.641765015625e+06, previous val  -7.980494309649e+00, real val 9.149566770786e+05, predicted val -9.782668451756e+13, rho_k -5.645037469093e-08
increasing alpha ratio or increasing minimum for quadratic coefficient
This step needs to be rejected.
alpd 1.000000000000e+06
convergence measure 2.053752313598e+00
function val convergence measure 9.782666022408e+13
iteration           objective                   residual                   step_size                   convg
8              5.556721692704e+06            9.782666022408e+13           6.274724053782e+00         2.053752313598e+00
solving full problem starts, iteration 8
real rval -7.980494309649e+00
previous base  4.641765015625e+06, current base 3.437693750000e+04, previous val  9.149566770786e+05, real val -7.980494309649e+00, predicted val -8.395505576262e+13, rho_k 6.577748442055e-08
increasing alpha ratio or increasing minimum for quadratic coefficient
This step needs to be rejected.
alpd 1.000000000000e-05
convergence measure 1.948976578885e+00
function val convergence measure 8.395508097105e+13
iteration           objective                   residual                   step_size                   convg
9              3.436895700569e+04            8.395508097105e+13           6.274724053782e+00         1.948976578885e+00
solving full problem starts, iteration 9
real rval 9.149566770786e+05
previous base  3.437693750000e+04, current base 4.641765015625e+06, previous val  -7.980494309649e+00, real val 9.149566770786e+05, predicted val -9.782668451756e+13, rho_k -5.645037469093e-08
increasing alpha ratio or increasing minimum for quadratic coefficient
This step needs to be rejected.
alpd 1.000000000000e+06
convergence measure 2.053752313598e+00
function val convergence measure 9.782666022408e+13
iteration           objective                   residual                   step_size                   convg
10              5.556721692704e+06            9.782666022408e+13           6.274724053782e+00         2.053752313598e+00
solving full problem starts, iteration 10
real rval -7.980494309649e+00
previous base  4.641765015625e+06, current base 3.437693750000e+04, previous val  9.149566770786e+05, real val -7.980494309649e+00, predicted val -8.395505576262e+13, rho_k 6.577748442055e-08
increasing alpha ratio or increasing minimum for quadratic coefficient
This step needs to be rejected.
alpd 1.000000000000e-05
convergence measure 1.948976578885e+00
function val convergence measure 8.395508097105e+13
iteration           objective                   residual                   step_size                   convg
11              3.436895700569e+04            8.395508097105e+13           6.274724053782e+00         1.948976578885e+00
solving full problem starts, iteration 11
real rval 9.149566770786e+05
previous base  3.437693750000e+04, current base 4.641765015625e+06, previous val  -7.980494309649e+00, real val 9.149566770786e+05, predicted val -9.782668451756e+13, rho_k -5.645037469093e-08
increasing alpha ratio or increasing minimum for quadratic coefficient
This step needs to be rejected.
alpd 1.000000000000e+06
convergence measure 2.053752313598e+00
function val convergence measure 9.782666022408e+13
iteration           objective                   residual                   step_size                   convg
12              5.556721692704e+06            9.782666022408e+13           6.274724053782e+00         2.053752313598e+00
solving full problem starts, iteration 12
real rval -7.980494309649e+00
previous base  4.641765015625e+06, current base 3.437693750000e+04, previous val  9.149566770786e+05, real val -7.980494309649e+00, predicted val -8.395505576262e+13, rho_k 6.577748442055e-08
increasing alpha ratio or increasing minimum for quadratic coefficient
This step needs to be rejected.
alpd 1.000000000000e-05
convergence measure 1.948976578885e+00
function val convergence measure 8.395508097105e+13
iteration           objective                   residual                   step_size                   convg
13              3.436895700569e+04            8.395508097105e+13           6.274724053782e+00         1.948976578885e+00
solving full problem starts, iteration 13
real rval 9.149566770786e+05
previous base  3.437693750000e+04, current base 4.641765015625e+06, previous val  -7.980494309649e+00, real val 9.149566770786e+05, predicted val -9.782668451756e+13, rho_k -5.645037469093e-08
increasing alpha ratio or increasing minimum for quadratic coefficient
This step needs to be rejected.
alpd 1.000000000000e+06
convergence measure 2.053752313598e+00
function val convergence measure 9.782666022408e+13
iteration           objective                   residual                   step_size                   convg
14              5.556721692704e+06            9.782666022408e+13           6.274724053782e+00         2.053752313598e+00
solving full problem starts, iteration 14
real rval -7.980494309649e+00
previous base  4.641765015625e+06, current base 3.437693750000e+04, previous val  9.149566770786e+05, real val -7.980494309649e+00, predicted val -8.395505576262e+13, rho_k 6.577748442055e-08
increasing alpha ratio or increasing minimum for quadratic coefficient
This step needs to be rejected.
alpd 1.000000000000e-05
convergence measure 1.948976578885e+00
function val convergence measure 8.395508097105e+13
iteration           objective                   residual                   step_size                   convg
15              3.436895700569e+04            8.395508097105e+13           6.274724053782e+00         1.948976578885e+00
solving full problem starts, iteration 15
real rval 9.149566770786e+05
previous base  3.437693750000e+04, current base 4.641765015625e+06, previous val  -7.980494309649e+00, real val 9.149566770786e+05, predicted val -9.782668451756e+13, rho_k -5.645037469093e-08
increasing alpha ratio or increasing minimum for quadratic coefficient
This step needs to be rejected.
alpd 1.000000000000e+06
convergence measure 2.053752313598e+00
function val convergence measure 9.782666022408e+13
iteration           objective                   residual                   step_size                   convg
16              5.556721692704e+06            9.782666022408e+13           6.274724053782e+00         2.053752313598e+00
solving full problem starts, iteration 16
real rval -7.980494309649e+00
previous base  4.641765015625e+06, current base 3.437693750000e+04, previous val  9.149566770786e+05, real val -7.980494309649e+00, predicted val -8.395505576262e+13, rho_k 6.577748442055e-08
increasing alpha ratio or increasing minimum for quadratic coefficient
This step needs to be rejected.
alpd 1.000000000000e-05
convergence measure 1.948976578885e+00
function val convergence measure 8.395508097105e+13
iteration           objective                   residual                   step_size                   convg
17              3.436895700569e+04            8.395508097105e+13           6.274724053782e+00         1.948976578885e+00
solving full problem starts, iteration 17
real rval 9.149566770786e+05
previous base  3.437693750000e+04, current base 4.641765015625e+06, previous val  -7.980494309649e+00, real val 9.149566770786e+05, predicted val -9.782668451756e+13, rho_k -5.645037469093e-08
increasing alpha ratio or increasing minimum for quadratic coefficient
This step needs to be rejected.
alpd 1.000000000000e+06
convergence measure 2.053752313598e+00
function val convergence measure 9.782666022408e+13
iteration           objective                   residual                   step_size                   convg
18              5.556721692704e+06            9.782666022408e+13           6.274724053782e+00         2.053752313598e+00
solving full problem starts, iteration 18
real rval -7.980494309649e+00
previous base  4.641765015625e+06, current base 3.437693750000e+04, previous val  9.149566770786e+05, real val -7.980494309649e+00, predicted val -8.395505576262e+13, rho_k 6.577748442055e-08
increasing alpha ratio or increasing minimum for quadratic coefficient
This step needs to be rejected.
alpd 1.000000000000e-05
convergence measure 1.948976578885e+00
function val convergence measure 8.395508097105e+13
iteration           objective                   residual                   step_size                   convg
19              3.436895700569e+04            8.395508097105e+13           6.274724053782e+00         1.948976578885e+00
solving full problem starts, iteration 19
real rval 9.149566770786e+05
previous base  3.437693750000e+04, current base 4.641765015625e+06, previous val  -7.980494309649e+00, real val 9.149566770786e+05, predicted val -9.782668451756e+13, rho_k -5.645037469093e-08
increasing alpha ratio or increasing minimum for quadratic coefficient
This step needs to be rejected.
alpd 1.000000000000e+06
convergence measure 2.053752313598e+00
function val convergence measure 9.782666022408e+13
iteration           objective                   residual                   step_size                   convg
20              5.556721692704e+06            9.782666022408e+13           6.274724053782e+00         2.053752313598e+00
solving full problem starts, iteration 20
real rval -7.980494309649e+00
previous base  4.641765015625e+06, current base 3.437693750000e+04, previous val  9.149566770786e+05, real val -7.980494309649e+00, predicted val -8.395505576262e+13, rho_k 6.577748442055e-08
increasing alpha ratio or increasing minimum for quadratic coefficient
This step needs to be rejected.

It will certainly be worth trying different Ipopt and HiOp PriDec options (and upgrading to a more recent version of HiOp).

ExaGO version

develop

System and environment details

See Frontier build config.

Additional information

No response

Proposed solution

No response

Start by running FUNCTIONALITY_TEST_SCOPFLOW_HIOP_TESTSUITE on Frontier and compare its output with the attached 300-second log, focusing on the HiOp PriDec iterations for case_ACTIVSg200.m. Trace why the test alternates between rejected steps and fails to converge; done means the SCOPFLOW functionality test converges and passes within a finite timeout.