File format time series support
- Dominant language
- Julia
- Stars
- 117
- Forks
- 19
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 36
Description
## Purpose
The current HDF5 file format is spatial only, and does not have any mechanism to denote time.
[Currently in ClimaAtmos](https://github.com/CliMA/ClimaAtmos.jl/blob/379acf2c4fad05f461df42a24f927be4fff06760/examples/hybrid/callbacks.jl#L358-L3660), we create a single file per save point: this has several downsides:
- we write the time value per file in an adhoc way (by calling `HDF5.write_attribute`)
- we duplicate the space data: each file writes the full domain, mesh, topology, space information
- when reloading data (e.g. for diagnostic purposes), each field will be on a distinct space (i.e. there is no mechanism to say that the spaces in two file are the same). In #747, I proposed a `FieldSeries` type, but this wasn't fully explained, and was not implemented.
This proposal aims to add time series support to our data format, so that full spatio-temporal data can be stored. There are two approaches I propose to support:
1. store everything in a single file (useful for small simulations and portability)
2. store the metadata (domain, mesh, topology, space) in one file, and the field data in a single file per time step.
We can unify these approaches by using [HDF5 virtual datasets (VDS)](https://portal.hdfgroup.org/display/HDF5/Introduction+to+the+Virtual+Dataset++-+VDS): in the 2nd case, the metadata file would also contain a virtual dataset for each spatio-temporal fields, where each time slice points to a HDF5 file containing the field data at that time point.
## Cost/benefits/risks
Benefits:
- it will give us a unified way of writing data over time, and make post-processing of data much easier.
- some small performance advantage in avoiding duplicated writing of metadata
Costs:
- Virtual dataset (VDS) support will require some additions to the Julia HDF5.jl package. As part of my exploratory work, I [opened a draft PR](https://github.com/JuliaIO/HDF5.jl/pull/1012), but this will need to be completed and merged as part of this work.
Risks:
- Virtual datasets (VDS) are a newer HDF5 feature, so there may be some teething issues. This should not present any problem for writing data (we will write the data directly to the source files, as the VDS interface does not create files), but there may be issues when reloading. Should this occur, we will also support a fallback interface for reading snapshots directly from files.
## Producers
TBD
## Components
The key components are:
- a file representation of sequence of discrete time points
- a file representation for spatio-temporal spaces
- expanding the field dataset layout types to support a trailing temporal dimension (e.g. `VIJFH` would become `VIJFHT`)
- interfaces for:
- defining a spatio-temporal fields in a file (with the number of time points potentially unknown)
- an interface for writing and reading slices of a spatio-temporal field at a given time point.
- a new writer object (`HDF5SplitWriter`) for writing metadata to a file, and fields to individual source files using the VDS feature.
## Inputs
### File layout
The overall principle would be to save the field data with an extra dimension, e.g. a sphere using a `VIJFH` layout, would be instead stored as a `VIJFHT` layout, with each slice in the trailing dimension being the time domain. Typically this will be an "unlimited" dimension, allowing unbounded append operations. The field would have an extra attribute `time_sequence`, which would be the name of the `TimeSequence`.
A `TimeSequence` objects are written as groups under a new `times` top-level group, which would contain three attributes
- `type`: `"TimeSequence"`
- `epoch`: is the time at the first step. This is either a floating point value (typically 0.0), or a string containing a [ISO8601](https://www.rfc-editor.org/rfc/rfc3339)-formatted datetime,
- `delta`: is the duration between time points.again, either a floating point value, or a string containing an ISO8601-formatted duration.
It may also contain an optional dataset:
- `values`: this is a 1D dataset containing the actual time values, as differences from the epoch. This accepts an optional `units` attribute (default = `"seconds"`)
**Note** that this SDI does _not_ propose adding relevant DataLayouts as Julia objects at this stage, but this could be done in future (e.g. for storing time series in memory).
This is designed so that the exact number of time points need not be known a priori, and can simply be determined by the length of the T dimension of a given field. In the case of a `HDF5SplitWriter` writer, this allows us to write the metadata file and close it before writing the field data.
When using a VDS approach, each source file would contain spatial fields in the same way it is currently (i.e. as a dataset under the `fields` group), but we would have an extra `space_file` attribute which describes the file the space is defined in.
### Interfaces
- A `TimeSequence` Julia object with two fields (matching the above file description), and corresponding write and read operations.
- `define_field(name::AbstractString, field::Union{Field, FieldVector}, timeseq::TimeSequnce)` would write the necessary metadata entries, and create a new dataset under `fields` but not write any data.
- in the case of `HDF5SplitWriter`, this would create the appropriate VDS mapping
- `append!(writer, name => field,...)`: this would write a new slice to each field
- `read_field(reader, name, timeidx::Integer)` read a field at a given time index
## Results and deliverables
- `HDF5Writer` will support writing time series to a single file
- `HDF5VDSWriter` will support writing time series to multiple files, with a single metadata file
## Task breakdown
A preliminary list of PRs and a preliminary timeline of PRs, milestones, and key results.
- [ ] Complete [HDF5.jl#1012](https://github.com/JuliaIO/HDF5.jl/pull/1012): requires addressing reviewer concerns adding necessary tests and docs
- [ ] Add mechanism to write
...
## Reviewers
The names of CliMA personnel who will review the Proposal and subsequent PRs.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.