Hydra for sampled performance testing?
Nobody has claimed this yet.
- Dominant language
- PLpgSQL
- Stars
- 1.6k
- Forks
- 358
- Avg merge
- 20h 30m
- Merged PRs (30d)
- 27
Description
I have a fantasy about defining a huge multidimensional space of test cases and for CI to explore this space with the resources that it has, for example with random sampling on each dimension. The idea is that the test cases would define many independent parameters and combinational explosion of these would need to be contained. This is inspired by property-based testing and the idea of treating testing as a search problem.
Fantasy details described in detail on SnabbCo/snabbswitch#688.
Is this something that one could reasonably do directly with Hydra? (Or otherwise could a separate test-driver process submit jobs to Hydra for execution and archiving?)
I would love to know more details about how the performance regression testing is done at LogicBlox as referenced in Rob Vermaas's NixConf talk.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the detailed proposal in SnabbCo/snabbswitch#688 and review how Hydra currently handles performance regression testing. Determine whether Hydra can sample multidimensional test cases and use available CI resources, or whether an external test driver would need to submit and archive jobs; document the feasible approach.
Written by the indexing model from the issue text.
Assessment
- Domain
- ci-cd, performance, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100