openpharma / openpharma/rbmi

Full test reproducibility appears to be failing on internal tests

Open
#482 10 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
R
Stars
21
Forks
12
PR merge metrics
No merged PRs in 30d

Description

https://statistical-engineering.pages.roche.com/-/systems-integration-tests/-/jobs/77276548/artifacts/public/logs/check-rbmi.html

  ── Failure ('test-reproducibility.R:75:9'): Results are Reproducible ───────────
  `drawobj` (`actual`) not equal to `drawobj2` (`expected`).
  
      actual$samples[[1]]$beta | expected$samples[[1]]$beta    
  [1] 4.844                    - 5.504                      [1]
  [2] 8.212                    - 6.047                      [2]
  [3] 0.724                    - -0.615                     [3]
  [4] 0.448                    - -0.599                     [4]
  [5] 3.090                    - 3.010                      [5]
  [6] 2.247                    - 2.715                      [6]
  [7] -1.141                   - 1.856                      [7]
  [8] -0.486                   - 2.180                      [8]
  
  actual$samples[[1]]$sigma$A vs expected$samples[[1]]$sigma$A
                                             [,1]        [,2]        [,3]
  - actual$samples[[1]]$sigma$A[1, ]    3.8996258  1.81540872  0.18346780
  + expected$samples[[1]]$sigma$A[1, ]  4.9890982  1.04694767 -0.13357694
  - actual$samples[[1]]$sigma$A[2, ]    1.8154087  1.88882220 -0.04483718
  + expected$samples[[1]]$sigma$A[2, ]  1.0469477  1.02407796 -0.03493389
  - actual$samples[[1]]$sigma$A[3, ]    0.1834678 -0.04483718  0.37916568
  + expected$samples[[1]]$sigma$A[3, ] -0.1335769 -0.03493389  0.35201823

Haven't checked but I think this is from the FULL test suite that doesn't get checked via the GitHub actions pipelines

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the linked check-rbmi.html artifact and test-reproducibility.R at line 75, then compare how the full test suite is run with the GitHub Actions pipelines. Reproduce the mismatch and identify whether the failure is limited to the full suite. Done means the reproducibility test passes consistently or the differing test coverage is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.