`ISO_8859_1` breaking UTF-8 in CLP `_logtype` column
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 55m
- Merged PRs (30d)
- 182
Description
# Cause
`clp-ffi-java` [internally use](https://github.com/y-scope/clp-ffi-java/blob/c4a74dbdeb09bd4e7e3d119826dddbe5005ccf53/src/main/java/com/yscope/clp/compressorfrontend/EncodedMessage.java#L30-L36) `StandardCharsets.ISO_8859_1` in `EncodedMessage.getLogTypeAsString();`

(`getDictionaryVarsAsStrings` also)
# Effect
https://github.com/apache/pinot/blob/0a4398634be81cdbbe891b3da249134ef98743e7/pinot-plugins/pinot-input-format/pinot-clp-log/src/main/java/org/apache/pinot/plugin/inputformat/clplog/CLPLogRecordExtractor.java#L151-L154
This makes some characters broken in `column_logtype` like this:
`Request processing failed: jakarta.validation.ConstraintViolationException: getAgentsList.from: /u0011 이상이어야 합니다`
into
`Request processing failed: jakarta.validation.ConstraintViolationException: getAgentsList.from: ì´ìì´ì´ì¼ í©ëë¤`
This is fine after going through the CLPDECODE function, but when dealing with individual `_logtype` columns, these broken strings don't seem appropriate (LIKE searches, etc).
The `clp-ffi-java` library makes all EncodedMessage member variables public. So it would be nice if pinot's `CLPLogMessageDecoder` could handle them (or at least match the other encodings used internally by pinot).
Contributor guide
Assessment
This issue has not been assessed yet.