Add plugins to support storing and querying JSON log events in Pinot
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 55m
- Merged PRs (30d)
- 182
Description
We want to be able to store JSON log events in Pinot so that they can be queried efficiently and so that we can reduce storage costs. Part of this involves encoding unstructured message fields in the log event using a new log compressor called [CLP](https://github.com/y-scope/clp). The other part is to transform the log event to fit a table's schema (e.g., extracting nested fields and storing them in a column). We think this can be done with a custom `StreamMessageDecoder` and a few UDFs.
We've written more about the motivation and proposal [here](https://docs.google.com/document/d/1nHZb37re4mUwEA258x3a2pgX13EWLWMJ0uLEDk1dUyU).
Contributor guide
Research direction
Start with the linked proposal and the CLP project, then locate Pinot's StreamMessageDecoder and UDF extension points mentioned in the issue. Done means plugins can encode unstructured JSON log fields, transform events to a table schema, and support efficient storage and querying in Pinot.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100