simonsobs / simonsobs/sotodlib

Something not working with preprocess+jobdb integration

Open
#1,734 0 comments 0 reactions 1 assignee View on GitHub

@mmccrackan is already working on this.

Since Sep 3, 2026.

preprocess
Dominant language
Python
Stars
19
Forks
23
Avg merge
1d 5h
Merged PRs (30d)
14

Description

I ran preprocess twice in a row with this command:

srun -n 32 --cpus-per-task=8 python -u \
    $HOME/software/sotodlib/sotodlib/site_pipeline/preprocess_tod.py \
    $HOME/software/so-preprocess-runs/20260824/satp1/preprocess_satp1_init_20260824.yaml \
    --query $HOME/software/so-preprocess-runs/20260824/satp1/satp1_test_obs_list.txt \
    --nproc 32 \
    --compress

I'm on the master branch and ran on the same obs_id list both times. Basically, I ran it once and it got half-way through before getting killed for time. On the second run it looks like the rank-1 thread crashed with:

Traceback (most recent call last):
  File "/global/homes/k/kmharrin/software/sotodlib/sotodlib/site_pipeline/preprocess_tod.py", line 499, in <module>
    main_launcher(main, get_parser)
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/global/homes/k/kmharrin/.local/soconda/20260812_0.2.5/lib/python3.13/site-packages/sotodlib/site_pipeline/utils/pipeline.py", line 24, in main_launcher
    return main_func(**vars(parser_func().parse_args(args=args)))
  File "/global/homes/k/kmharrin/software/sotodlib/sotodlib/site_pipeline/preprocess_tod.py", line 480, in main
    _main(executor=executor,
    ~~~~~^^^^^^^^^^^^^^^^^^^
          as_completed_callable=as_completed_callable,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<13 lines>...
          raise_error=raise_error,
          ^^^^^^^^^^^^^^^^^^^^^^^^
          pb_path=pb_path)
          ^^^^^^^^^^^^^^^^
  File "/global/homes/k/kmharrin/software/sotodlib/sotodlib/site_pipeline/preprocess_tod.py", line 353, in _main
    j.mark_visited()

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.