apache / apache/pinot

Refactor duplicate function-evaluation paths in ExpressionTransformer and CustomFunctionEnricher

Open
#18,216 0 comments 0 reactions 0 assignees View on GitHub
cleanup ingestion java priority: low refactor
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 3h
Merged PRs (30d)
195

Description

### Motivation
`pinot-segment-local/src/main/java/org/apache/pinot/segment/local/recordtransformer/ExpressionTransformer.java` and `pinot-segment-local/src/main/java/org/apache/pinot/segment/local/recordtransformer/enricher/function/CustomFunctionEnricher.java` still carry reciprocal TODOs to merge the two implementations.

Both code paths evaluate function-based transformations during ingestion, but they live behind different extension points (`RecordTransformer` vs `RecordEnricher`). That makes behavior drift and duplicate maintenance more likely.

### Scope
- Document the overlap and intended ownership between the two implementations.
- Consolidate shared evaluation logic behind one path or a shared abstraction.
- Preserve current ingestion behavior for transform configs and enricher configs.
- Add or update focused tests for parity and backward compatibility.

### Notes
Observed on `upstream/master` on April 15, 2026.

Contributor guide

Open the contributing guide

Research direction

Start with ExpressionTransformer.java and CustomFunctionEnricher.java, reading the reciprocal TODOs and tracing their RecordTransformer and RecordEnricher extension points. Compare how both paths evaluate function-based transformations, then add or update focused parity and backward-compatibility tests for transform and enricher configurations. Done means shared ownership is clear, duplicate evaluation logic is consolidated, and current ingestion behavior is preserved.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, data-engineering
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.