datafusion-contrib / datafusion-contrib/StreamFusion

Native IF: admit BOOLEAN result branches

Open
#210 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
120
Forks
8
Avg merge
7h 29m
Merged PRs (30d)
76

Description

Rechecked on 2026-09-20 against canonical main 89da01cdad7342c4d825e36eaf4e8dc73bc7af9f, released Apache Flink 2.2.1, JDK 17, default compatibility settings. Each expression was run against independent stock-Flink and StreamFusion-enabled streaming environments with bounded runtime DataStream input (parallelism 1), so source values cannot be folded away. Results and resolved result types agree through Flink fallback; this is a native support gap, not a demonstrated wrong-result bug.

PR #169 closed #164 for its verified result-type scope. BOOLEAN result branches remain unsupported although released Flink executes them.

Runtime schema and row: a BOOLEAN=true, i INT=3, b BOOLEAN=false.

SELECT IF(i < 5, a, b) FROM src;

Stock Flink returns TRUE with BOOLEAN result type. StreamFusion returns the same result through zero native substitutions and Calc: IF requires a verified scalar result type.

Independent runtime condition probes also confirm TRUE/FALSE/NULL select TRUE/FALSE/FALSE respectively, all through the same fallback.

This comes from a runtime adaptation of ScalarFunctionsTest. This ticket concerns the remaining BOOLEAN overload, not the already-shipped STRING/numeric IF coverage.

Cover TRUE/FALSE/NULL conditions, nullable BOOLEAN branches, nested predicates, unselected failing branches, and an all-filtered batch. Verify released Flink's actual BOOLEAN overload/typing contract before changing the whitelist.

Suggested priority: P2, basic scalar control-flow coverage.

Acceptance: add runtime-source SQL parity tests that assert native Calc execution as well as values and resolved types; include NULLs and boundary/error cases; retain safe fallback for unverified forms and update docs/operators/calc-filter.md with the admitted scope.

Latest-main validation: rebuilt the debug native library from this commit and ran 562 targeted runtime SQL cases covering these support gaps and adjacent controls. This issue group contains 4 successful comparisons that still record explicit fallback. Host-rejected/both-error cases are excluded from that count. This validates released Flink 2.2.1; it does not claim execution of the Flink 1.18 profile. No implementation is included.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the runtime adaptation of ScalarFunctionsTest and the existing native Calc IF coverage, then verify Flink 2.2.1's BOOLEAN overload and typing contract. Add runtime-source SQL parity tests for the listed conditions, nullability, nesting, errors, and filtering, asserting native Calc execution, values, and resolved types. Update docs/operators/calc-filter.md with the admitted scope while retaining fallback for unverified forms.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
data-engineering, documentation, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.