Macs3 runs out of memory due to multiprocessing when run from a python script.

Thank you for this great tool, snapATAC2 has been extremely helpful for my multiome analyses.

There appears to be an issue with how multiprocessing is implemented in snapatac2.tl.macs3() that is revealed when this function is run from within a python script. When I run this in an interactive code environment (Jupyter), there are no issues.

However, if this is run from the command line inside a .py script, new instances of the main script are re-launched and quickly consumes all memory available. This is not an issue with available memory. I tried running snapatac2.tl.macs3() protected by

if __name__ == "__main__":
  snapatac2.tl.macs3()

and this did not work either.

For example, here is a script that fails to run properly from the command line, and the print statement will be re-printed 8 times:

import pandas as pd
import scanpy as sc
import muon as mu
import snapatac2 as snap
import numpy as np

print('Calling peaks with macs')

mdata = mu.read_h5mu('path/to/data.h5mu')

snap.tl.macs3(mdata['atac'], groupby='Cell Type', n_jobs=8)

Would you please be able to help fix this? For now, I can just run macs3 from jupyter. But it would be better to be able to have this run in the background with a script for large datasets.

Thank you,

Ethan

Start at the snapatac2.tl.macs3() entry point and trace how multiprocessing is launched when called from a command-line Python script. Reproduce the provided example with n_jobs=8, compare it with the Jupyter case, and confirm that the main script is not re-launched and memory remains bounded.