Error related to parallelism when running `snap.tl.macs3` when running on the cluster
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 323
- Forks
- 43
- PR merge metrics
- No merged PRs in 30d
Description
Recently I had a weird issue when running snap.tl.macs3 on a cluster, my script would get stuck for a while and then throw an error related to parallelism issues:
0%| | 0/12 [00:19<?, ?it/s]
Traceback (most recent call last):
File "/mnt/sds-hd/sd22b002/projects/GRETA/greta_benchmark/callpeaks.py", line 18, in <module>
snap.tl.macs3(adata, groupby='cell_type', n_jobs=n_jobs, tempdir=tempdir)
File "/opt/conda/envs/env/lib/python3.10/site-packages/snapatac2/tools/_call_peaks.py", line 155, in macs3
peaks = _par_map(_call_peaks, [(x,) for x in fragments.values()], n_jobs)
File "/opt/conda/envs/env/lib/python3.10/site-packages/snapatac2/tools/_call_peaks.py", line 221, in _par_map
raise RuntimeError("Some worker process has died unexpectedly.")
RuntimeError: Some worker process has died unexpectedly.
Despite this, the same script would work in my local machine.
In case someone has the same issue, I've found that the solution is to wrap the call to macs3 with the __main__ conditional statement. Here is an example:
import snapatac2 as snap
if __name__ == '__main__':
# Read data
adata = snap.read(snap.datasets.pbmc5k(type='annotated_h5ad'), backed=None)
# Subset to make things faster
msk = adata.obs.groupby('cell_type', observed=False).head(50).index
adata = adata[msk, :].copy()
# Call ATAC-seq peaks using MACS
snap.tl.macs3(adata, groupby='cell_type', n_jobs=8)
print('Done!')
Maybe this could also be added in the docs.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Review the documentation for the snap.tl.macs3 entry point and determine where its cluster or parallelism usage is explained. Document that calls using parallel workers should be protected by an main conditional, including the example from the issue; done means users can find and follow the workaround.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- bioinformatics
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100