desihub / desihub/desispec

dailyops pipeline crash on zero-sized test file

Open
#2,153 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
42
Forks
26
Avg merge
1d 14h
Merged PRs (30d)
18

Description

20231226/0021376/desi-00213760.fits.fz is a bogus zero-sized FITS file that the desiops pipeline crashed on. At the same time, it was a test exposure with a valid request file that we could have parsed to know to skip it:

  “SEQUENCE”: “Spectrographs”,
  “FLAVOR”: “zero”,
  “OBSTYPE”: “other”,
  “NOADJUSTMENTS”: true,

@akremin in the pipeline refactor, let's test this case and immediately skip over the exposure if the request*.json file is OBSTYPE=other. If there is an actual use case for needing to read the raw data file even for test exposures, please describe that here and we'll sort out a way to be more robust to this.

Current failure from ~desi/daily/logfiles/daily-20231226.log

##################### 213760 #########################
INFO:exptable.py:437:summarize_exposure: Summarizing exposure: 213760
INFO:exptable.py:451:summarize_exposure: Using user specified colnames and coldefaults.
INFO:exptable.py:505:summarize_exposure: Found request file: /global/cfs/cdirs/desi/spectro/data/20231226/00213760/request-00213760.json
INFO:exptable.py:549:summarize_exposure: Found raw data file: /global/cfs/cdirs/desi/spectro/data/20231226/00213760/desi-00213760.fits.fz
Traceback (most recent call last):
  File "/global/common/software/desi/perlmutter/desiconda/20230111-2.1.0/code/desispec/main/bin/desi_daily_proc_manager", line 165, in <module>
    daily_processing_manager(specprod=args.specprod, exp_table_path=args.exp_table_path,
  File "/global/common/software/desi/perlmutter/desiconda/20230111-2.1.0/code/desispec/main/py/desispec/scripts/daily_processing.py", line 261, in daily_processing_manager
    erow = summarize_exposure(path_to_data, night, exp, expobstypes, colnames, coldefaults, verbosely=False)
  File "/global/common/software/desi/perlmutter/desiconda/20230111-2.1.0/code/desispec/main/py/desispec/workflow/exptable.py", line 550, in summarize_exposure
    dat_header, fx = load_raw_data_header(pathname=datpath, return_filehandle=True)
  File "/global/common/software/desi/perlmutter/desiconda/20230111-2.1.0/code/desispec/main/py/desispec/workflow/desi_proc_funcs.py", line 211, in load_raw_data_header
    fx = fitsio.FITS(pathname)
  File "/global/common/software/desi/perlmutter/desiconda/20230111-2.1.0/conda/lib/python3.10/site-packages/fitsio/fitslib.py", line 520, in __init__
    self._FITS = _fitsio_wrap.FITS(filename, self.intmode, create)
OSError: FITSIO status = 107: tried to move past end of file
ffopen could not interpret primary array header of file: 
/global/cfs/cdirs/desi/spectro/data/20231226/00213760/desi-00213760.fits.fz

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in py/desispec/workflow/exptable.py at summarize_exposure, and trace how the request*.json file is parsed before the raw data header is loaded through load_raw_data_header in desi_proc_funcs.py. Reproduce the zero-sized FITS case from the daily log and verify that an exposure whose request has OBSTYPE="other" is skipped without attempting to open the raw data file.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.