desihub / desihub/desispec

improve fiberflat_vs_humidity I/O efficiency

Open
#1,900 0 comments 0 reactions 0 assignees View on GitHub
performance
Dominant language
Python
Stars
42
Forks
26
Avg merge
1d 14h
Merged PRs (30d)
18

Description

desi_compute_fiberflat_vs_humidity appears to be more sensitive to NERSC I/O performance than other steps, sometimes leading to job timeouts. Some of this is worse than it used to be due to the gzipped frame files now being inefficient to read headers. From a quick look, some ideas to improve this:

* in scripts/proc.py, the gzipped frame file header is read but apparently not used in the "desi_compute_fiberflat_vs_humidity" block, and then re-read later in the sky subtraction block.
* in scripts.humidity_corrected_fiberflat.py, the header is read again separately, even though it is also read as part of `read_frame` (it is the `frame.meta`)
* I don't think fiberflat_vs_humidity uses the resolution matrix; if that's the case the I/O could also be faster using `read_frame(filename, skip_resolution=True)`

I'm flagging this for Iron since fluctuations in the timing of this step in test runs led to job timeouts (we'll also increase the job time to give more robustness).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the desi_compute_fiberflat_vs_humidity block in scripts/proc.py and the header reads in scripts.humidity_corrected_fiberflat.py. Trace header use through sky subtraction and read_frame/frame.meta, then verify whether humidity correction needs the resolution matrix before considering skip_resolution=True. Done means reduced redundant I/O with unchanged pipeline results and improved timing; the issue names no validation tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, performance
Issue type
Refactor
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.