simonsobs / simonsobs/sotodlib
Something not working with preprocess+jobdb integration
Open
@mmccrackan is already working on this.
Since Sep 3, 2026.
preprocess
- Dominant language
- Python
- Stars
- 19
- Forks
- 23
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 14
Description
I ran preprocess twice in a row with this command:
srun -n 32 --cpus-per-task=8 python -u \
$HOME/software/sotodlib/sotodlib/site_pipeline/preprocess_tod.py \
$HOME/software/so-preprocess-runs/20260824/satp1/preprocess_satp1_init_20260824.yaml \
--query $HOME/software/so-preprocess-runs/20260824/satp1/satp1_test_obs_list.txt \
--nproc 32 \
--compress
I'm on the master branch and ran on the same obs_id list both times. Basically, I ran it once and it got half-way through before getting killed for time. On the second run it looks like the rank-1 thread crashed with:
Traceback (most recent call last):
File "/global/homes/k/kmharrin/software/sotodlib/sotodlib/site_pipeline/preprocess_tod.py", line 499, in <module>
main_launcher(main, get_parser)
~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
File "/global/homes/k/kmharrin/.local/soconda/20260812_0.2.5/lib/python3.13/site-packages/sotodlib/site_pipeline/utils/pipeline.py", line 24, in main_launcher
return main_func(**vars(parser_func().parse_args(args=args)))
File "/global/homes/k/kmharrin/software/sotodlib/sotodlib/site_pipeline/preprocess_tod.py", line 480, in main
_main(executor=executor,
~~~~~^^^^^^^^^^^^^^^^^^^
as_completed_callable=as_completed_callable,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...<13 lines>...
raise_error=raise_error,
^^^^^^^^^^^^^^^^^^^^^^^^
pb_path=pb_path)
^^^^^^^^^^^^^^^^
File "/global/homes/k/kmharrin/software/sotodlib/sotodlib/site_pipeline/preprocess_tod.py", line 353, in _main
j.mark_visited()
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.