Caliper Support for test cases
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 10
- Forks
- 6
- PR merge metrics
- No merged PRs in 30d
Description
Adding caliper support to the test suites that use ATS requires updates to each existing test case, which can be onerous. Since all codes wanting caliper output need to do this we'd like it to be an option in ATS.
Current implementation using ATS introspection:
#ATS:if checkGlue("caliper"):
#ATS: name = "TestCase2D" + str( uuid4() )
#ATS: outputdirectory=log.directory+'/caliperoutput/'+name+'/'+name+'.cali'
#ATS: myExe = manager.options.executable + ''' --caliper "spot(output=%s)" '''%outputdirectory
#ATS: t = test(executable=myExe, clas="%s radGroups=16 steps=5 useBC=False meshtype=polygonalrz runDirBaseName=%s" % (SELF,name), nn=1, np=ndomains, nt=nthreads, ngpu=ngpus, suite="threads", label='caliper 80a 16g rz FP regression')
What the option would do is:
- If caliper, Append **--caliper spot(output=)** to the test executable. Note that this didn't work as expected when passed as a **clas**, only when inserted into the executable var (myExe above).
- The tests **output directory** needs to be unique to the test object so that after introspection each individual test has it's own directory. For example, a caliper dir could be saved with the same name as the test log directory, but extension _cal_ or something like that. Perhaps at the same level, so that caliper data is preserved while run logs can still be cleaned up. We don't want to have to keep _everything_ , just the caliper data.
- That naming scheme would also be ideal for archiving caliper data for visualization making it easier to point SPOT at the cumulative runs for an individual test case.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing ATS option handling and the test object’s output-directory creation, then compare it with the inline ATS introspection example in the issue. Implement the caliper option so each test gets a unique preserved output directory and verify it with an ATS test suite that uses executable arguments.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing-qa, tooling
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100