stan-dev / stan-dev/performance-tests-cmdstan

Stan using the same seed=1234 | strange performance tests

Open
#3 0 comments 0 reactions 1 assignee View on GitHub

@seantalts is already working on this.

Since Apr 14, 2019.

Dominant language
Python
Stars
9
Forks
8
PR merge metrics
No merged PRs in 30d

Description

method = sample (Default)
  sample
    num_samples = 1000 (Default)
    num_warmup = 1000 (Default)
    save_warmup = 0 (Default)
    thin = 1 (Default)
    adapt
      engaged = 1 (Default)
      gamma = 0.050000000000000003 (Default)
      delta = 0.80000000000000004 (Default)
      kappa = 0.75 (Default)
      t0 = 10 (Default)
      init_buffer = 75 (Default)
      term_buffer = 50 (Default)
      window = 25 (Default)
    algorithm = hmc (Default)
      hmc
        engine = nuts (Default)
          nuts
            max_depth = 10 (Default)
        metric = diag_e (Default)
        metric_file =  (Default)
        stepsize = 1 (Default)
        stepsize_jitter = 0 (Default)
id = 0 (Default)
data
  file = XXX
init = 2 (Default)
random
  seed = 1234
output
  file = XXX
  diagnostic_file =  (Default)
  refresh = 100 (Default)

Tests I found during Jenkins runs that give strange performance reports on the same machine in subsequent runs.

PrevBuildURI90% | CurrentBuildURI90% | RelativeDiff | RelativeDiff(%)

286               40526               40240           14069.93%
example-models/bugs_examples/vol1/pump/pump.data.R
---
443               42348               41905            9459.37%
example-models.bugs_examples.vol1.dyes.dyes  
---
23237                 533              -22704             -97.71%
example-models.bugs_examples.vol1.equiv.equiv  
---
8302               39174               30872             371.86%
example-models.bugs_examples.vol1.leuk.leuk 
---
7807                2274               -5533             -70.87%
example-models.bugs_examples.vol1.mice.mice 
---
163                 365                 202             123.93%
example-models.bugs_examples.vol2.beetles.beetles_cloglog   
---
7807               39129               31322             401.20%
example-models.bugs_examples.vol1.mice.mice
---
23237                 506              -22731             -97.82%
example-models.bugs_examples.vol1.equiv.equiv
---
 153              114870              114717           74978.43%
example-models.bugs_examples.vol2.beetles.beetles_probit

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.