dailyops pipeline crash on zero-sized test file
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 42
- Forks
- 26
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 18
Description
20231226/0021376/desi-00213760.fits.fz is a bogus zero-sized FITS file that the desiops pipeline crashed on. At the same time, it was a test exposure with a valid request file that we could have parsed to know to skip it:
“SEQUENCE”: “Spectrographs”,
“FLAVOR”: “zero”,
“OBSTYPE”: “other”,
“NOADJUSTMENTS”: true,
@akremin in the pipeline refactor, let's test this case and immediately skip over the exposure if the request*.json file is OBSTYPE=other. If there is an actual use case for needing to read the raw data file even for test exposures, please describe that here and we'll sort out a way to be more robust to this.
Current failure from ~desi/daily/logfiles/daily-20231226.log
##################### 213760 #########################
INFO:exptable.py:437:summarize_exposure: Summarizing exposure: 213760
INFO:exptable.py:451:summarize_exposure: Using user specified colnames and coldefaults.
INFO:exptable.py:505:summarize_exposure: Found request file: /global/cfs/cdirs/desi/spectro/data/20231226/00213760/request-00213760.json
INFO:exptable.py:549:summarize_exposure: Found raw data file: /global/cfs/cdirs/desi/spectro/data/20231226/00213760/desi-00213760.fits.fz
Traceback (most recent call last):
File "/global/common/software/desi/perlmutter/desiconda/20230111-2.1.0/code/desispec/main/bin/desi_daily_proc_manager", line 165, in <module>
daily_processing_manager(specprod=args.specprod, exp_table_path=args.exp_table_path,
File "/global/common/software/desi/perlmutter/desiconda/20230111-2.1.0/code/desispec/main/py/desispec/scripts/daily_processing.py", line 261, in daily_processing_manager
erow = summarize_exposure(path_to_data, night, exp, expobstypes, colnames, coldefaults, verbosely=False)
File "/global/common/software/desi/perlmutter/desiconda/20230111-2.1.0/code/desispec/main/py/desispec/workflow/exptable.py", line 550, in summarize_exposure
dat_header, fx = load_raw_data_header(pathname=datpath, return_filehandle=True)
File "/global/common/software/desi/perlmutter/desiconda/20230111-2.1.0/code/desispec/main/py/desispec/workflow/desi_proc_funcs.py", line 211, in load_raw_data_header
fx = fitsio.FITS(pathname)
File "/global/common/software/desi/perlmutter/desiconda/20230111-2.1.0/conda/lib/python3.10/site-packages/fitsio/fitslib.py", line 520, in __init__
self._FITS = _fitsio_wrap.FITS(filename, self.intmode, create)
OSError: FITSIO status = 107: tried to move past end of file
ffopen could not interpret primary array header of file:
/global/cfs/cdirs/desi/spectro/data/20231226/00213760/desi-00213760.fits.fz
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in py/desispec/workflow/exptable.py at summarize_exposure, and trace how the request*.json file is parsed before the raw data header is loaded through load_raw_data_header in desi_proc_funcs.py. Reproduce the zero-sized FITS case from the daily log and verify that an exposure whose request has OBSTYPE="other" is skipped without attempting to open the raw data file.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100