apache / apache/pinot

Persist raw data from kafka topic as it is

Open
#8,399 9 comments 0 reactions 0 assignees View on GitHub
feature
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 3h
Merged PRs (30d)
195

Description

**Feature request:**
To save raw data from topic in pinot table.

**Use case :**
We have lots of complex schemas and we are using pinot for saving and retreiving topic data with times stamp and some other fields.
We do not want to map all nested columns from complex schema and create pinot schema and use lots of transformation functions. There are some places we want raw data as it is in pinot table.

**Sample data :**

`{
"header": {
"tid": "12wee",
"rid": 1,
"timestamp": 1647347092337
},
"status": "200_SUCCESS",
"jasData": {
"sdata": -22.89122,
"cnn": 0.823469,
"kli": 2.238848,
"olp": [
{
"ovPerc": 0.032486767,
"hg": 30.0,
"abshi": 6.661863
}
],
"terrkl": {
"ovPerc": 0.9675132,
"dist": [
-25.17232,
-25.17232,
-25.130081
]
},
"bcut": 2.77
},
"rgData": {
"pre": 102033.33,
"pv": 0.16,
"t": 287.36,
"timestamp": 1647347069000
},
"timestamp": 1647347092337
}
}`

Contributor guide

Open the contributing guide

Research direction

The issue names no source file, test, or entry point. Start by tracing Pinot's Kafka ingestion and schema-handling paths, then establish the supported raw-data behavior and corresponding tests for the sample payload before considering the feature complete.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.