E3SM-Project / E3SM-Project/scorpio

Very slow IO time when writing sub-hourly data

Open
#168 1 comment 0 reactions 1 assignee Claimed by @jayeshkrishna View on GitHub
Dominant language
C++
Stars
22
Forks
20
Avg merge
12d 19m
Merged PRs (30d)
1

Description

When sub-hourly data is turned on in CESM2, the write call gets progressively slower. There is what looks like a linear cost term that is based on the number of calls to PIO_write_darray. I believe it is due to the use of a single linked list in the subroutine "add_data_to_buffer" in the piodarray.F90.in file. Because each write is so small when generating sub-hourly data, the linked list file%data_list_top gets to be very long before the buffer data is actually written to disk. A reduction in the pio_buffer_size_limit appears to significantly increase IO performance because it reduces the maximum length of the linked list. Replacing a single linked list with a double linked list should resolve this issue while still maximizing the size of IO to disk.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.