Need to improve molpro memory trsh

Warning: Troubleshooting iC3H6CNOOH job sp_a32590 which failed with status: "errored,"
with keywords: ['NoOutput']
in molpro.
The error "Log file could not be read" was derived from the following line in the log file:
"".
Troubleshooting sp job in molpro for iC3H6CNOOH using shift
Running job sp_a32594 for iC3H6CNOOH
Currently running jobs:
{'iC3H6CNOOH': ['scan_a32587', 'freq_a32589', 'scan_a32591', 'scan_a32592', 'scan_a32593', 'sp_a32594']}
  Ending job freq_a32589 for iC3H6CNOOH (run time: 0:11:45)
  Ending job sp_a32594 for iC3H6CNOOH (run time: 0:25:50)


Warning: Troubleshooting iC3H6CNOOH job sp_a32594 which failed with status: "errored,"
with keywords: ['Memory']
in molpro.
The error "Additional memory required: 135.10 MW" was derived from the following line in the log file:
" A further 135.10 Mwords of memory are needed for the triples to run. Increase memory to 359.38 Mwords.".
Troubleshooting sp job in molpro for iC3H6CNOOH using memory: 20.5625 GB instead of 14 GB
Running job sp_a32595 for iC3H6CNOOH
  Ending job sp_a32595 for iC3H6CNOOH (run time: 0:21:48)


Warning: Troubleshooting iC3H6CNOOH job sp_a32595 which failed with status: "errored,"
with keywords: ['Memory']
in molpro.
The error "Additional memory required: 30.10 MW" was derived from the following line in the log file:
" A further 30.10 Mwords of memory are needed for the triples to run. Increase memory to 359.38 Mwords.".
Troubleshooting sp job in molpro for iC3H6CNOOH using memory: 26.34375 GB instead of 20.5625 GB
Running job sp_a32596 for iC3H6CNOOH
Currently running jobs:
{'iC3H6CNOOH': ['scan_a32587', 'scan_a32591', 'scan_a32592', 'scan_a32593', 'sp_a32596']}
Currently running jobs:
{'iC3H6CNOOH': ['scan_a32587', 'scan_a32591', 'scan_a32592', 'scan_a32593', 'sp_a32596']}

The first job returned an empty output.out file. The output file on the server was empty as well, and the Slurm message was:

0:Child process terminated prematurely, status=: 0
(rank:0 hostname:node05 pid:28293):ARMCI DASSERT fail. src/common/signaltrap.c:SigChldHandler():178 cond:0
application called MPI_Abort(comm=0x84000007, -1) - process 0

The other jobs ran fine, but molpro repeatedly asked for more memory. Perhaps a better and simpler strategy for memory trsh would be to increase job_mem to the max allowed by the server if the ESS requests for more (the quantity it asks for might be increased down the road, so don't try predicting how much it needs). If it fails again due to memory, try changing server if the ESS is available on more than one server, but do so only if the other server has more memory allocation.

Another solution would be to teach ARC to parallelize over several nodes.

Start by tracing the Molpro troubleshooting path that handles Memory and NoOutput failures, then inspect how job_mem limits and ESS server availability are represented. The issue provides no file or test entry points, so identify the relevant implementation and tests before changing behavior. Done means memory requests can reach the server limit and server switching respects available memory.