Tutorial on Cheaha system testing

What would you like to see added?

A common thread among office hours visits and tickets for low-level fixes for pre-existing software on Cheaha is long turn-around times. A researcher will come in to get help with a low-level bug, error or failure in their workflow. Sometimes its the software, sometimes its a change in the module system, sometimes its a new parameter, sometimes its a new dataset with corruption or errors or unexpected features.

In most cases we are able to troubleshoot the issue, but oftentimes the researcher will only have a full-sized dataset to verify that a troubleshooting step had an observable effect. In the genomics space this can mean hours or days of processing, only to find out the step had no effect. This can lead to drawn-out troubleshooting sessions, or spread out over a month, with lots of frame-shifting, reminders of where we are, etc.

These sessions could be made substantially shorter if researchers were aware of the concept of, and prepared for, black box system testing. The general idea behind black box system testing is to have a known set of inputs that produce a known set of outputs. Then when we change the black box (troubleshoot), we verify that the outputs haven't changed. In many cases, the inputs can be made very small so that the black box processes more quickly than for typical inputs. Having a prepared "test set" of inputs would make the troubleshooting sessions much faster. It would also enhance reproducibility and trust in the code, which are scientific benefits.

An article which looks at various software testing types at a high level is: https://www.softwaretestinghelp.com/types-of-software-testing/. We may also consider rewriting the article for our docs in the context of scientific and research computing.

Start by reviewing the linked overview of software testing and the issue's Cheaha troubleshooting examples. Develop a tutorial explaining black-box system testing for scientific and research computing, including small known inputs and expected outputs; done means the guidance is adapted to this documentation project's audience and context.