Bad draw error when running chunks
- Dominant language
- Python
- Stars
- 25
- Forks
- 6
- PR merge metrics
- No merged PRs in 30d
Description
I ran into an issue when fitting multiple single features in a chunked fashion. The chains finish but there seems to be an error with the draw dimensions. I tried the errant feature individually and it worked as intended so there may be some file-system/HPC issue at hand. This issue does not occur for all chunks so my guess is it is not related to the BIRDMAn code directly.
```python
for feature_num in range(start_num, stop_num):
print(f"Feature: {feature_num}")
if feature_num > num_feats:
print("Over the table size")
exit()
feature_id = fids[feature_num]
outdir = f"/panfs/grahman/birdman-analyses/speed/outdir/{feature_num}_{feature_id}"
os.makedirs(outdir, exist_ok=True)
model = NegativeBinomialSingle(...)
model.compile_model()
model.fit_model(sampler_args={"output_dir": outdir})
inf = model.to_inference_object()
print(inf.posterior)
print(az.loo(inf, pointwise=True))
```
```
INFO:cmdstanpy:found newer exe file, not recompiling
INFO:cmdstanpy:compiled model file: /home/grahman/miniconda3/envs/birdman-benchmarking/lib/python3.7/site-packages/birdman/templates/negative_binomial_single
INFO:cmdstanpy:start chain 1
INFO:cmdstanpy:start chain 2
INFO:cmdstanpy:start chain 3
INFO:cmdstanpy:start chain 4
INFO:cmdstanpy:finish chain 4
INFO:cmdstanpy:finish chain 3
INFO:cmdstanpy:finish chain 2
INFO:cmdstanpy:finish chain 1
Traceback (most recent call last):
File "src/speed/run_birdman_chunked.py", line 46, in
model.fit_model(sampler_args={"output_dir": tmpdirname})
File "/home/grahman/miniconda3/envs/birdman-benchmarking/lib/python3.7/site-packages/birdman/model_base.py", line 164, in fit_model
**sampler_args
File "/home/grahman/miniconda3/envs/birdman-benchmarking/lib/python3.7/site-packages/cmdstanpy/model.py", line 835, in sample
mcmc = CmdStanMCMC(runset, validate_csv, logger=self._logger)
File "/home/grahman/miniconda3/envs/birdman-benchmarking/lib/python3.7/site-packages/cmdstanpy/stanfit.py", line 433, in __init__
self.validate_csv_files()
File "/home/grahman/miniconda3/envs/birdman-benchmarking/lib/python3.7/site-packages/cmdstanpy/stanfit.py", line 700, in validate_csv_files
thin=self._thin,
File "/home/grahman/miniconda3/envs/birdman-benchmarking/lib/python3.7/site-packages/cmdstanpy/utils.py", line 464, in check_sampler_csv
meta = scan_sampler_csv(path, is_fixed_param)
File "/home/grahman/miniconda3/envs/birdman-benchmarking/lib/python3.7/site-packages/cmdstanpy/utils.py", line 520, in scan_sampler_csv
lineno = scan_sampling_iters(fd, dict, lineno)
File "/home/grahman/miniconda3/envs/birdman-benchmarking/lib/python3.7/site-packages/cmdstanpy/utils.py", line 781, in scan_sampling_iters
lineno, num_cols, len(line.split(','))
ValueError: line 80: bad draw, expecting 71969 items, found 73695
deleting tmpfiles dir: /tmp/tmptufa9zy5
done
```
cc @mortonjt
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the chunked run from src/speed/run_birdman_chunked.py and compare it with the individually successful feature. Inspect model_base.py and the generated CmdStan output around the reported bad draw to determine whether chunking or the filesystem/HPC environment causes the column mismatch. Done means identifying a reproducible cause and confirming whether the malformed draw is produced by BIRDMAn or its execution environment.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- bioinformatics
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100