gchq / gchq/sleeper

Extend range of fields accepted by using Arrow types rather than Sleeper's internal types

Open
#576 0 comments 0 reactions 0 assignees View on GitHub
epic
Dominant language
Java
Stars
107
Forks
29
Avg merge
19h 46m
Merged PRs (30d)
141

Description

We want to allow richer data to be stored in Sleeper, specifically data with fields which consist of structs, arbitrarily nested structs, etc. Both Parquet and Arrow support this, and as we store data in Parquet it should be possible to store more complex data than Sleeper currently supports. Therefore this proposal is to switch away from using Sleeper's current built-in types such as IntType, MapType, etc to Arrow types. Sleeper needs to know which are the row key and sort fields, and those fields need to be comparable, and so it is not possible to just use an Arrow schema. At the very least we need the additional information about which fields are the row and sort fields.

The challenge with this is that the parquet-mr library does not support reading records into Arrow RecordBatches. Therefore we believe it is best to attempt this epic after we have switched to using DataFusion for data processing (see https://github.com/gchq/sleeper/issues/1388).

Related to:
- https://github.com/gchq/sleeper/issues/6059
- https://github.com/gchq/sleeper/issues/6184

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.