apache / apache/pinot

Add plugins to support storing and querying JSON log events in Pinot

Open
#9,819 7 comments 5 reactions 0 assignees View on GitHub
design-review feature PEP-Request
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

We want to be able to store JSON log events in Pinot so that they can be queried efficiently and so that we can reduce storage costs. Part of this involves encoding unstructured message fields in the log event using a new log compressor called [CLP](https://github.com/y-scope/clp). The other part is to transform the log event to fit a table's schema (e.g., extracting nested fields and storing them in a column). We think this can be done with a custom `StreamMessageDecoder` and a few UDFs.

We've written more about the motivation and proposal [here](https://docs.google.com/document/d/1nHZb37re4mUwEA258x3a2pgX13EWLWMJ0uLEDk1dUyU).

Contributor guide

Open the contributing guide

Research direction

Start with the linked proposal and the CLP project, then locate Pinot's StreamMessageDecoder and UDF extension points mentioned in the issue. Done means plugins can encode unstructured JSON log fields, transform events to a table schema, and support efficient storage and querying in Pinot.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.