Investigate tests with very long run times
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 839
- Forks
- 220
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 14
Description
Description
A few tests in our non-prob unit tests take an extraordinarily long amount of time to run.
We currently run 5005 tests in the Full Unit tests on Jenkins. 3343 of those run for less than a millisecond, 4418 less than 10 ms, 4760 less than 100ms and 4902 less than a second.
In total, the tests run for around 13-14 minutes, with the 100 tests that individually run for 1 or longer taking 12 minutes of the 13-14. This seems a bit surprising to me and I think we should investigate this. 13 minutes isn't a lot, but if we are wasting 10 minutes of those, I think it's worth taking a bit to not waste computationally cycles.
We might as well discover that all of these are needed.
Listing the top 15 tests that take almost 8 minutes to run in total:
| Test | Links | Time [s] |
|---|---|---|
| choleskyDecomposeGeneral | link | 68 |
| choleskyDecomposeGeneralBig | link | 62 |
| lub_stdvec_lb_vec_ub_mat_constrain | link | 51 |
| offset_multiplier_stdvec_mu_vec_sigma_mat_constrain | link | 37 |
| complexPow | link | 35 |
| lub_stdvec_lb_mat_ub_vec_constrain | link | 31 |
| lub_stdvec_mat_scalar1_constrain | link | 27 |
| 25 | ||
| fallingFactorial_vec | link | 23 |
| offset_multiplier_stdvec_mu_mat_sigma_vec_constrain | link | 22 |
| offset_multiplier_stdvec_mat_scalar1_constrain | link | 19 |
| lub_stdvec_mat_scalar_constrain | link | 17 |
| matrixExpMultiply | link | 14 |
| 13 | ||
| offset_multiplier_stdvec_mat_scalar_constrain | link | 12 |
All of these are tests in mix/.My feeling is that the constrain tests and Cholesky decomposition tests are unnecessarily big and could be easily reduced. ODEs are more difficult problems and my gut feeling there is that is expected.
Current Version:
v4.2.1
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the Full Unit test timings and inspect the listed slow tests in test/unit/math/mix/fun/, especially cholesky_decompose_test.cpp, the constrain test files, pow_part2_test.cpp, falling_factorial_test.cpp, and matrix_exp_multiply_test.cpp. Determine whether the large cases are necessary and record which tests can be reduced without losing coverage; done means a justified change and improved timings.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- performance, testing-qa
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100