Expiremental Apache Parquet time series store
@MikeNeilson is already working on this.
Since Aug 31, 2026.
- Dominant language
- Java
- Stars
- 28
- Forks
- 25
- Avg merge
- 4d 22h
- Merged PRs (30d)
- 20
Description
Is your feature request related to a problem? Please describe.
Storing data to Oracle, in small batches, it consuming excessive resources.
Additionally having all reads for time series go through oracle has limit parallelism.
Describe the solution you'd like
Time Series data is a good candidate for Column store approaches. Apache Parquet is a format, with several libraries, and higher level abstractions, available.
Additionally the source and destination are abstracted. Parquet files can be written to disk or S3, or other.
Under this particular issue a write buffer, and if reasonable read cache (e.g. store on read), should be created that just uses local disk storage as an initial write location.
A background process will review the existing files read, and store them to the database
NOTE: this will be behind some form of feature flag (I will likely allow a header to override so I can test in cloud environments) as we don't want this behavior getting to prod yet... except maybe as the cache (see additional context.)
Describe alternatives you've considered
Redis, Custom formats to S3, DynamoDb
Additional context
None of the alternatives are mutually exclusive.
The data-entry-date may not match as we don't currently allow the api to let user data set that... however, this isn't user data. For the moment, not going to worry about it.
It will likely be better to just write data to s3 parquet files vs the database itself; however, that can't be used in production until other WM depenant services also write through CDA.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.