datafusion-contrib / datafusion-contrib/StreamFusion
Native GREATEST/LEAST: support runtime STRING, mixed exact numerics and TIMESTAMP/LTZ
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 120
- Forks
- 8
- Avg merge
- 7h 29m
- Merged PRs (30d)
- 76
Description
Rechecked on 2026-09-20 against canonical main 89da01cdad7342c4d825e36eaf4e8dc73bc7af9f, released Apache Flink 2.2.1, JDK 17, default compatibility settings. Each expression was run against independent stock-Flink and StreamFusion-enabled streaming environments with bounded runtime DataStream input (parallelism 1), so source values cannot be folded away. Results and resolved result types agree through Flink fallback; this is a native support gap, not a demonstrated wrong-result bug.
Runtime extrema still fall back for common scalar inputs. Execute each expression separately over the indicated runtime columns:
| Input types/values | Expression | Flink result | Native rejection |
|---|---|---|---|
| i INT=2, d DECIMAL(12,2)=1.25 | GREATEST(i,d) / LEAST(i,d) | 2.00 / 1.25, DECIMAL(12,2) | operand types require unverified coercion |
| a STRING='abc', b STRING='def' | GREATEST(a,b) / LEAST(a,b) | 'def' / 'abc' | requires ASCII-provable string operands |
| a TIMESTAMP(9)='1969-12-31 23:59:59.999999999', b TIMESTAMP(9)='2026-01-01 00:00:00' | GREATEST(a,b) | b | supports integers, strings, boolean, and matching decimals |
| a TIMESTAMP_LTZ(9)=Instant(-1 second + 999999999 ns), b=Instant(1 second) | LEAST(a,b) | 1969-12-31T23:59:59.999999999Z | same type gate |
Example: SELECT GREATEST(a,b) FROM src. The STRING restriction triggers even when the actual runtime strings are ASCII. Adapted GreatestLeastFunctionsITCase SQL and independent minimal probes both confirm the gaps.
Preserve Flink coercion, decimal precision/scale, NULL propagation and UTF-16 string ordering, including supplementary characters. Preserve full timestamp precision/range and LTZ behavior across session zones. Existing same-type admitted forms should remain covered as controls.
Suggested priority: P2, immediately after exact numeric scalar coverage.
Acceptance: add runtime-source SQL parity tests that assert native Calc execution as well as values and resolved types; include NULLs and boundary/error cases; retain safe fallback for unverified forms and update docs/operators/calc-filter.md with the admitted scope.
Latest-main validation: rebuilt the debug native library from this commit and ran 562 targeted runtime SQL cases covering these support gaps and adjacent controls. This issue group contains 13 successful comparisons that still record explicit fallback. Host-rejected/both-error cases are excluded from that count. This validates released Flink 2.2.1; it does not claim execution of the Flink 1.18 profile. No implementation is included.
Related: PR #44 documented narrower admitted overloads and the remaining fallback; this ticket tracks the unimplemented forms.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the adapted GreatestLeastFunctionsITCase SQL cases and the independent runtime probes described in the issue, then inspect native Calc execution for GREATEST/LEAST. Use runtime-source inputs to cover STRING, mixed exact numerics, TIMESTAMP and TIMESTAMP_LTZ, including NULLs and boundaries. Done means native Calc tests match Flink values and types, safe fallback remains for unverified forms, and docs/operators/calc-filter.md records the admitted scope.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- stream-processing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100