S3ResultsOutput options
- Dominant language
- Java
- Stars
- 107
- Forks
- 29
- Avg merge
- 19h 46m
- Merged PRs (30d)
- 141
Description
`S3ResultsOutput` needs several properties to construct the `ParquetWriter`. These are: the compression codec to use, the row group size, the page size, whether to use dictionary encoding for row keys, sort keys, value fields. The row group size and page size are found by looking in the provided `Map config`, and if they are not present then the values are taken from the instance properties `DEFAULT_RESULTS_ROW_GROUP_SIZE`, `DEFAULT_RESULTS_PAGE_SIZE`. The compression codec is taken from the config and if not present it is taken from the table properties. The values of the dictionary fields are taken from the table properties `DICTIONARY_ENCODING_FOR_ROW_KEY_FIELDS`, `DICTIONARY_ENCODING_FOR_SORT_KEY_FIELDS`, `DICTIONARY_ENCODING_FOR_VALUE_FIELDS`.
This should be standardised. All these properties should be first set from the value in the config. If not present they should be taken from table properties that are specific to results writing (i.e. table property equivalents of `DEFAULT_RESULTS_ROW_GROUP_SIZE`). Those properties should have default values set by default instance properties analogous to `DEFAULT_DICTIONARY_ENCODING_FOR_ROW_KEY_FIELDS`.
Contributor guide
Assessment
This issue has not been assessed yet.