apache / apache/pinot

[PEP-Request] ThriftMessageDecoder implementation for thrift stream ingestion

Open
#10,414 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

_I tested this feature in local, and want to contribute to this project._

**1. What needs to be done?**
: Implement "ThriftMessageDecoder" for Thrift Stream Ingestion

**2. Why the feature is needed (e.g. describing the use case).**
: Here is example case
- I work on AI team and our services communicate based on thrift.
- We stack log data on kafka.
- I want to stream ingest thrift data through kafka, and add OLAP dashboards.

**3. It may also include an initial idea/proposal on how as well.**
- locate thrift-jar file to /lib folder
- using ThriftRecordExtractor & complexTypeConfig for decoding

Contributor guide

Open the contributing guide

Research direction

Locate the Thrift stream-ingestion entry points and inspect how ThriftRecordExtractor and complexTypeConfig are used for decoding. Check the proposed thrift JAR placement under /lib and existing Kafka ingestion tests; done means ThriftMessageDecoder can decode the described Thrift records for stream ingestion and the relevant tests pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
data-engineering, stream-processing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.