desihub / desihub/desispec

desispec.io.emlinefit.write_emlines should use a temporary file

Open Beginner friendly
#1,997 0 comments 0 reactions 0 assignees View on GitHub
cleanup
Dominant language
Python
Stars
42
Forks
26
Avg merge
1d 21h
Merged PRs (30d)
15

Description

`desispec.io.emlinefit.write_emlines` currently doesn't use a temporary filename when writing, leading to the failure case of leaving a correctly-named file behind with corrupted contents if the job times out during writing. See #1996 for cases of this happening in iron.

Update `desispec.io.emlinefit.write_emlines` to follow the tempfile pattern used elsewhere, which writes to a temporary filename first, and only after successful completion it renames the file to the final output filename:
```
from desispec.io.util import get_tempfilename
tmpfile = get_tempfilename(outfile)
# write to "tmpfile" instead of "outfile", then
os.rename(tmpfile, outfile)
```

More info: `get_tempfilename` creates a temporary filename in the same output directory while avoiding some common problems:
* `tmpfile` is in same dir as `outfile` so that the rename is ~instantaneous, not copying from local temp to final location
* `tmpfile` uses the same extension as `outfile` so that things like `Table.write` can still auto-derive the format to use
* it treats `.fits.gz` and `.fits.fz` like a single extension to preserve
* `tmpfile` names are unique across MPI ranks (mostly; it uses the PID so could have multi-node unlucky PID collisions, but it is still way better than just f"{outfile}.tmp"

We'll clean this up by hand for iron, but should do this before the next major prod

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading desispec.io.emlinefit.write_emlines and the get_tempfilename implementation in desispec.io.util. Trace where the output filename is passed to the writer, then verify that writing uses the temporary path and renames it to the requested output only after successful completion.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.