improve fiberflat_vs_humidity I/O efficiency
- Dominant language
- Python
- Stars
- 42
- Forks
- 26
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 18
Description
desi_compute_fiberflat_vs_humidity appears to be more sensitive to NERSC I/O performance than other steps, sometimes leading to job timeouts. Some of this is worse than it used to be due to the gzipped frame files now being inefficient to read headers. From a quick look, some ideas to improve this:
* in scripts/proc.py, the gzipped frame file header is read but apparently not used in the "desi_compute_fiberflat_vs_humidity" block, and then re-read later in the sky subtraction block.
* in scripts.humidity_corrected_fiberflat.py, the header is read again separately, even though it is also read as part of `read_frame` (it is the `frame.meta`)
* I don't think fiberflat_vs_humidity uses the resolution matrix; if that's the case the I/O could also be faster using `read_frame(filename, skip_resolution=True)`
I'm flagging this for Iron since fluctuations in the timing of this step in test runs led to job timeouts (we'll also increase the job time to give more robustness).
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the desi_compute_fiberflat_vs_humidity block in scripts/proc.py and the header reads in scripts.humidity_corrected_fiberflat.py. Trace header use through sky subtraction and read_frame/frame.meta, then verify whether humidity correction needs the resolution matrix before considering skip_resolution=True. Done means reduced redundant I/O with unchanged pipeline results and improved timing; the issue names no validation tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, performance
- Issue type
- Refactor
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100