E3SM-Project / E3SM-Project/scorpio
Very slow IO time when writing sub-hourly data
- Dominant language
- C++
- Stars
- 22
- Forks
- 20
- Avg merge
- 12d 19m
- Merged PRs (30d)
- 1
Description
When sub-hourly data is turned on in CESM2, the write call gets progressively slower. There is what looks like a linear cost term that is based on the number of calls to PIO_write_darray. I believe it is due to the use of a single linked list in the subroutine "add_data_to_buffer" in the piodarray.F90.in file. Because each write is so small when generating sub-hourly data, the linked list file%data_list_top gets to be very long before the buffer data is actually written to disk. A reduction in the pio_buffer_size_limit appears to significantly increase IO performance because it reduces the maximum length of the linked list. Replacing a single linked list with a double linked list should resolve this issue while still maximizing the size of IO to disk.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.