apache / apache/gluten

[FLINK] Support common stateless scalar expressions

Open
#12,665 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Description

## Motivation

The velox backend already implements a set of common Flink SQL scalar functions, but the gluten-flink planner has no mapping from Flink function name to velox Substrait operator, so these functions cannot be offloaded to velox today.

Functions in scope:

- Time fields: `YEAR`, `MONTH`, `DAYOFMONTH`, `DAYOFYEAR`, `DAYOFWEEK`, `QUARTER`, `HOUR`, `MINUTE`, `SECOND`, `WEEK`
- String/control: `SUBSTRING`, `COALESCE`, `FROM_BASE64`, `JSON_VALUE`

Additionally, `RexNodeConverter` does not convert TIMESTAMP literals to `TimestampValue.create(epochSeconds, nanos)`, so any literal-TIMESTAMP-bearing expression fails earlier in the conversion path.

## Proposed Capability

The planner should produce a valid Substrait plan for SQL like:

```sql
SELECT
EXTRACT(HOUR FROM ts),
SUBSTRING(name, 1, 3),
COALESCE(name, 'unknown'),
JSON_VALUE(payload, '$.user.id')
FROM ...
```

### Gluten version

main branch

Contributor guide

Open the contributing guide

Research direction

Start by tracing the gluten-flink planner's function-name mapping into Velox Substrait operators, then read the RexNodeConverter timestamp-literal conversion path. Verify coverage for the listed time, string, control, and JSON_VALUE expressions, plus TIMESTAMP literals. Done means the example SQL produces valid Substrait plans and literal-timestamp expressions no longer fail during conversion.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, sql
Domain
backend, data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.