ESCOMP / ESCOMP/CTSM

run_sys_tests: Allow running multiple create_test commands on a single batch node (useful for izumi)

Open
#1,742 0 comments 1 reaction 0 assignees View on GitHub
testing
Dominant language
Fortran
Stars
352
Forks
361
Avg merge
6d 6h
Merged PRs (30d)
8

Description

On izumi, the batch system is set up to limit the number of jobs to no more than 8 per user. We test with 4 compilers on izumi (though this will soon be reduced to 3), meaning that 4 (soon 3) of our 8 jobs are tied up as the create_test build jobs. Making matters worse, we specify the `--retry` option on izumi so that failed tests are rerun; an implication of this is that the build job doesn't exit until all tests finish. This means that a given user can really only have 4 (soon to be 5) test jobs running at once. This impacts turnaround time when running the test suite and makes it impossible to run two instances of the test suite at once on izumi – which I sometimes want to do. (If you tried to kick off two test suites simultaneously, I believe you'd be stuck in a situation where all 8 of your allowed jobs would be tied up in build jobs that are waiting for their submitted tests to finish, but the submitted tests won't get to run until the build jobs are eventually kicked out due to hitting their wallclock limits.)

I think the situation would be improved on izumi if we submitted a single batch job that kicked off all 4 (soon 3) create_test jobs in the background. These 4 jobs would contend for resources on the one compute node, but I think the situation will still be better than it is now, since you could run 7 rather than 4 tests simultaneously, and that is the limiting factor right now.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.