biocore / biocore/BIRDMAn

Bad draw error when running chunks

Open
#59 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
25
Forks
6
PR merge metrics
No merged PRs in 30d

Description

I ran into an issue when fitting multiple single features in a chunked fashion. The chains finish but there seems to be an error with the draw dimensions. I tried the errant feature individually and it worked as intended so there may be some file-system/HPC issue at hand. This issue does not occur for all chunks so my guess is it is not related to the BIRDMAn code directly.

```python
for feature_num in range(start_num, stop_num):
print(f"Feature: {feature_num}")
if feature_num > num_feats:
print("Over the table size")
exit()
feature_id = fids[feature_num]
outdir = f"/panfs/grahman/birdman-analyses/speed/outdir/{feature_num}_{feature_id}"
os.makedirs(outdir, exist_ok=True)

model = NegativeBinomialSingle(...)
model.compile_model()
model.fit_model(sampler_args={"output_dir": outdir})

inf = model.to_inference_object()
print(inf.posterior)
print(az.loo(inf, pointwise=True))

```

```
INFO:cmdstanpy:found newer exe file, not recompiling
INFO:cmdstanpy:compiled model file: /home/grahman/miniconda3/envs/birdman-benchmarking/lib/python3.7/site-packages/birdman/templates/negative_binomial_single
INFO:cmdstanpy:start chain 1
INFO:cmdstanpy:start chain 2
INFO:cmdstanpy:start chain 3
INFO:cmdstanpy:start chain 4
INFO:cmdstanpy:finish chain 4
INFO:cmdstanpy:finish chain 3
INFO:cmdstanpy:finish chain 2
INFO:cmdstanpy:finish chain 1
Traceback (most recent call last):
File "src/speed/run_birdman_chunked.py", line 46, in
model.fit_model(sampler_args={"output_dir": tmpdirname})
File "/home/grahman/miniconda3/envs/birdman-benchmarking/lib/python3.7/site-packages/birdman/model_base.py", line 164, in fit_model
**sampler_args
File "/home/grahman/miniconda3/envs/birdman-benchmarking/lib/python3.7/site-packages/cmdstanpy/model.py", line 835, in sample
mcmc = CmdStanMCMC(runset, validate_csv, logger=self._logger)
File "/home/grahman/miniconda3/envs/birdman-benchmarking/lib/python3.7/site-packages/cmdstanpy/stanfit.py", line 433, in __init__
self.validate_csv_files()
File "/home/grahman/miniconda3/envs/birdman-benchmarking/lib/python3.7/site-packages/cmdstanpy/stanfit.py", line 700, in validate_csv_files
thin=self._thin,
File "/home/grahman/miniconda3/envs/birdman-benchmarking/lib/python3.7/site-packages/cmdstanpy/utils.py", line 464, in check_sampler_csv
meta = scan_sampler_csv(path, is_fixed_param)
File "/home/grahman/miniconda3/envs/birdman-benchmarking/lib/python3.7/site-packages/cmdstanpy/utils.py", line 520, in scan_sampler_csv
lineno = scan_sampling_iters(fd, dict, lineno)
File "/home/grahman/miniconda3/envs/birdman-benchmarking/lib/python3.7/site-packages/cmdstanpy/utils.py", line 781, in scan_sampling_iters
lineno, num_cols, len(line.split(','))
ValueError: line 80: bad draw, expecting 71969 items, found 73695
deleting tmpfiles dir: /tmp/tmptufa9zy5
done
```

cc @mortonjt

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the chunked run from src/speed/run_birdman_chunked.py and compare it with the individually successful feature. Inspect model_base.py and the generated CmdStan output around the reported bad draw to determine whether chunking or the filesystem/HPC environment causes the column mismatch. Done means identifying a reproducible cause and confirming whether the malformed draw is produced by BIRDMAn or its execution environment.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
bioinformatics
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.