Reduce IO operations by using flush instead of repeated open/close.
- Dominant language
- Fortran
- Stars
- 249
- Forks
- 82
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 3
Description
This was suggested by Nick Carriero of the Flatiron Scientific Computing Core. Nick wrote to me with:
> One quick thing to do would be to use flush rather than close, if the goal is to ensure data is written to disk. The metadata ops involved in closing and then reopening a file are much "heavier" than just reading from or writing to a file.
This came up because I was running many jobs on a cluster and the number of ops became an issue. Nick looked into it, noticed that 3/7 of those ops were involved in opening and re-closing `history.data`, `profiles.index`, and `pgstar.dat`, each of which gets opened then closed once per step.
His suggestion is to open these files once at the start of a run, then use the Fortran `FLUSH` intrinsic after our write statements are all done to ensure that all data is written to disk. This can happen once per step, and we just close the file handle at the very end.
I don't think this is urgent, and I have a workaround for my use case (just locating all of the files on a local scratch system then copying them back when the run finishes), but I didn't want to forget it because it seems like a good idea for a rainy day.
Contributor guide
Research direction
Find the Fortran code that opens and closes history.data, profiles.index, and pgstar.dat, then trace when those operations occur during each step. Keep the files open for the run, flush them after each step's writes, and close the handles at the end; verify that output remains complete and usable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- fortran
- Domain
- hpc, performance
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100