gchq / gchq/sleeper

Add ability to write data to Rfiles instead of Parquet files

Open
#27 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
107
Forks
29
Avg merge
19h 46m
Merged PRs (30d)
141

Description

Add a table property that controls whether data is written to Rfiles or Parquet files. Add a writer and a reader that uses Rfiles. This will require concatenating fields (e.g. if there are two row key fields then their values will need to be concatenated into the row key in the Rfile). There is code in Gaffer to do this.

This issue should be done after https://github.com/gchq/sleeper/issues/26.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.