Persist raw data from kafka topic as it is
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 195
Description
**Feature request:**
To save raw data from topic in pinot table.
**Use case :**
We have lots of complex schemas and we are using pinot for saving and retreiving topic data with times stamp and some other fields.
We do not want to map all nested columns from complex schema and create pinot schema and use lots of transformation functions. There are some places we want raw data as it is in pinot table.
**Sample data :**
`{
"header": {
"tid": "12wee",
"rid": 1,
"timestamp": 1647347092337
},
"status": "200_SUCCESS",
"jasData": {
"sdata": -22.89122,
"cnn": 0.823469,
"kli": 2.238848,
"olp": [
{
"ovPerc": 0.032486767,
"hg": 30.0,
"abshi": 6.661863
}
],
"terrkl": {
"ovPerc": 0.9675132,
"dist": [
-25.17232,
-25.17232,
-25.130081
]
},
"bcut": 2.77
},
"rgData": {
"pre": 102033.33,
"pv": 0.16,
"t": 287.36,
"timestamp": 1647347069000
},
"timestamp": 1647347092337
}
}`
Contributor guide
Research direction
The issue names no source file, test, or entry point. Start by tracing Pinot's Kafka ingestion and schema-handling paths, then establish the supported raw-data behavior and corresponding tests for the sample payload before considering the feature complete.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100