desispec.io.emlinefit.write_emlines should use a temporary file
- Dominant language
- Python
- Stars
- 42
- Forks
- 26
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 15
Description
`desispec.io.emlinefit.write_emlines` currently doesn't use a temporary filename when writing, leading to the failure case of leaving a correctly-named file behind with corrupted contents if the job times out during writing. See #1996 for cases of this happening in iron.
Update `desispec.io.emlinefit.write_emlines` to follow the tempfile pattern used elsewhere, which writes to a temporary filename first, and only after successful completion it renames the file to the final output filename:
```
from desispec.io.util import get_tempfilename
tmpfile = get_tempfilename(outfile)
# write to "tmpfile" instead of "outfile", then
os.rename(tmpfile, outfile)
```
More info: `get_tempfilename` creates a temporary filename in the same output directory while avoiding some common problems:
* `tmpfile` is in same dir as `outfile` so that the rename is ~instantaneous, not copying from local temp to final location
* `tmpfile` uses the same extension as `outfile` so that things like `Table.write` can still auto-derive the format to use
* it treats `.fits.gz` and `.fits.fz` like a single extension to preserve
* `tmpfile` names are unique across MPI ranks (mostly; it uses the PID so could have multi-node unlucky PID collisions, but it is still way better than just f"{outfile}.tmp"
We'll clean this up by hand for iron, but should do this before the next major prod
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading desispec.io.emlinefit.write_emlines and the get_tempfilename implementation in desispec.io.util. Trace where the output filename is passed to the writer, then verify that writing uses the temporary path and renames it to the requested output only after successful completion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100