USACE / USACE/cwms-data-api

Expiremental Apache Parquet time series store

Open
#1,914 0 comments 0 reactions 1 assignee View on GitHub

@MikeNeilson is already working on this.

Since Aug 31, 2026.

enhancement
Dominant language
Java
Stars
28
Forks
25
Avg merge
4d 22h
Merged PRs (30d)
20

Description

Is your feature request related to a problem? Please describe.
Storing data to Oracle, in small batches, it consuming excessive resources.
Additionally having all reads for time series go through oracle has limit parallelism.

Describe the solution you'd like
Time Series data is a good candidate for Column store approaches. Apache Parquet is a format, with several libraries, and higher level abstractions, available.

Additionally the source and destination are abstracted. Parquet files can be written to disk or S3, or other.

Under this particular issue a write buffer, and if reasonable read cache (e.g. store on read), should be created that just uses local disk storage as an initial write location.
A background process will review the existing files read, and store them to the database

NOTE: this will be behind some form of feature flag (I will likely allow a header to override so I can test in cloud environments) as we don't want this behavior getting to prod yet... except maybe as the cache (see additional context.)

Describe alternatives you've considered
Redis, Custom formats to S3, DynamoDb

Additional context
None of the alternatives are mutually exclusive.

The data-entry-date may not match as we don't currently allow the api to let user data set that... however, this isn't user data. For the moment, not going to worry about it.

It will likely be better to just write data to s3 parquet files vs the database itself; however, that can't be used in production until other WM depenant services also write through CDA.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.