gchq / gchq/sleeper

S3ResultsOutput options

Open
#686 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
107
Forks
29
Avg merge
19h 46m
Merged PRs (30d)
141

Description

`S3ResultsOutput` needs several properties to construct the `ParquetWriter`. These are: the compression codec to use, the row group size, the page size, whether to use dictionary encoding for row keys, sort keys, value fields. The row group size and page size are found by looking in the provided `Map config`, and if they are not present then the values are taken from the instance properties `DEFAULT_RESULTS_ROW_GROUP_SIZE`, `DEFAULT_RESULTS_PAGE_SIZE`. The compression codec is taken from the config and if not present it is taken from the table properties. The values of the dictionary fields are taken from the table properties `DICTIONARY_ENCODING_FOR_ROW_KEY_FIELDS`, `DICTIONARY_ENCODING_FOR_SORT_KEY_FIELDS`, `DICTIONARY_ENCODING_FOR_VALUE_FIELDS`.

This should be standardised. All these properties should be first set from the value in the config. If not present they should be taken from table properties that are specific to results writing (i.e. table property equivalents of `DEFAULT_RESULTS_ROW_GROUP_SIZE`). Those properties should have default values set by default instance properties analogous to `DEFAULT_DICTIONARY_ENCODING_FOR_ROW_KEY_FIELDS`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.