Extend range of fields accepted by using Arrow types rather than Sleeper's internal types
- Dominant language
- Java
- Stars
- 107
- Forks
- 29
- Avg merge
- 19h 46m
- Merged PRs (30d)
- 141
Description
We want to allow richer data to be stored in Sleeper, specifically data with fields which consist of structs, arbitrarily nested structs, etc. Both Parquet and Arrow support this, and as we store data in Parquet it should be possible to store more complex data than Sleeper currently supports. Therefore this proposal is to switch away from using Sleeper's current built-in types such as IntType, MapType, etc to Arrow types. Sleeper needs to know which are the row key and sort fields, and those fields need to be comparable, and so it is not possible to just use an Arrow schema. At the very least we need the additional information about which fields are the row and sort fields.
The challenge with this is that the parquet-mr library does not support reading records into Arrow RecordBatches. Therefore we believe it is best to attempt this epic after we have switched to using DataFusion for data processing (see https://github.com/gchq/sleeper/issues/1388).
Related to:
- https://github.com/gchq/sleeper/issues/6059
- https://github.com/gchq/sleeper/issues/6184
Contributor guide
Assessment
This issue has not been assessed yet.