apache / apache/pinot

[Feature] support type inference and automatic type casting when searching for ScalarFunction

Open
#8,605 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

Background
===
Currently Pinot already implements some of the type hoisting - for example:
* the datatable builder automatically converts int/float/long/double numeric type content to the corresponding data schema required data type.

However sometime this doesn't work quite well. for example
* sum(long, long) will result in double

Propose
===
- Create a type inference system to automatically hoist types to when searching for a matching `@ScalarFunction`. for example `sum(int, long)` should match `sum(long, long)` with an implicit type cast from `int` to `long` for the first argument.
- implementing a type casting scheme using a bitmap approach:
- for example using numeric values, `double, float, long, int, short, byte` corresponding to a single bit in a byte value (`1<<6` ~ `1<<0`) and make all function arguments to mask into the byte and uses the highest non-zero bit corresponding type as the numeric type to cast to.
- for other type we can have different rules to match types
- implicit type cast methods can also be unified and reused in other part of the query logic and ingestion.

Contributor guide

Open the contributing guide

Research direction

Start by examining the existing ScalarFunction lookup and the datatable builder's numeric type handling. Define how type inference should match calls such as sum(int, long) to sum(long, long), including implicit casts and rules for non-numeric types. Done means matching functions can apply the intended casts and the casting approach can be reused in query logic and ingestion.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.