E3SM-Project / E3SM-Project/scorpio

ADIOS flush

Open
#228 0 comments 0 reactions 2 assignees Claimed by @dqwu View on GitHub
ADIOS enhancement High Priority Next Release
Dominant language
C++
Stars
22
Forks
20
Avg merge
12d 19m
Merged PRs (30d)
1

Description

Currently ADIOS caches all data written out by the user and writes it out to ADIOS BP files when the file is closed by the user. ADIOS currently ignores PIO_syncfile() calls from the user and does not maintain a buffer limit for the cached user data.

Using this scheme, ADIOS could potentially run out of memory. This issue can be solved by,

* Supporting sync calls
* Maintaining a buffer cache limit and internally syncing data to BP files

Currently data written to the BP files cannot be read back until the file is closed, and converted to NetCDF (Unlike other iotypes supported by PIO where data written out can be read back after a PIO_syncfile() call). So we might need to add an optional flag to PIO_syncfile() to indicate that the user does not intend to read back the data and the data can be stored/synced by ADIOS (other iotypes can ignore this flag).

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.