Native STRING helpers: track FROM_BASE64, numeric SP... | datafusion-contrib/StreamFusion

Nobody has claimed this yet.

Rechecked on 2026-09-20 against canonical main 89da01cdad7342c4d825e36eaf4e8dc73bc7af9f, released Apache Flink 2.2.1, JDK 17, default compatibility settings. Each expression was run against independent stock-Flink and StreamFusion-enabled streaming environments with bounded runtime DataStream input (parallelism 1), so source values cannot be folded away. Results and resolved result types agree through Flink fallback; this is a native support gap, not a demonstrated wrong-result bug.

Several less central scalar STRING overloads remain unsupported. Grouped here as a follow-up checklist after arithmetic, comparisons and string predicates. Adapted ScalarFunctionsTest SQL and independent runtime probes confirm each item.

Execute each projection separately over runtime columns:

Input	Projection	Flink result	Rejection
s STRING='aGVsbG8='	FROM_BASE64(s)	'hello', STRING	unsupported function/operator: FROM_BASE64
s STRING='AQIDBA=='	SPLIT_INDEX(s,73,0)	'AQ', STRING	SPLIT_INDEX requires STRING, STRING, INT arguments
s STRING='test', bits INT=256	SHA2(s,bits)	9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08, VARCHAR(128)	SHA2 requires a literal bit length of 224, 256, 384, or 512
s STRING='123'	IS_DECIMAL(s) / IS_DIGIT(s) / IS_ALPHA(s)	TRUE / TRUE / FALSE	unsupported function/operator: corresponding name
s STRING='abc'	IS_DECIMAL(s) / IS_DIGIT(s) / IS_ALPHA(s)	FALSE / FALSE / TRUE	same

Example: SELECT FROM_BASE64(s) FROM src. Each tested expression returns equal results via zero native substitutions. Classification also tested '-12.50', empty STRING and NULL: all three return FALSE for empty/NULL and declare BOOLEAN NOT NULL.

FROM_BASE64: preserve the released STRING output contract, binary-input overload, invalid encoding and NULL behavior.
SPLIT_INDEX numeric separator overload: the integer is a character code, not the text representation of the number; 256 and 0 return NULL in the executed cases.
SHA2: allow verified per-row bit lengths; preserve NULL/unsupported-size behavior.
IS_DECIMAL / IS_DIGIT / IS_ALPHA: preserve accepted syntax, Unicode classification and non-nullable FALSE for NULL.

Suggested priority: P2, after the common exact-numeric and STRING filtering gaps.

Acceptance: add runtime-source SQL parity tests that assert native Calc execution as well as values and resolved types; include NULLs and boundary/error cases; retain safe fallback for unverified forms and update docs/operators/calc-filter.md with the admitted scope.

Latest-main validation: rebuilt the debug native library from this commit and ran 562 targeted runtime SQL cases covering these support gaps and adjacent controls. This issue group contains 29 successful comparisons that still record explicit fallback. Host-rejected/both-error cases are excluded from that count. This validates released Flink 2.2.1; it does not claim execution of the Flink 1.18 profile. No implementation is included.

Related: PR #32 supports literal SHA2 widths; PR #44 leaves FROM_BASE64 and the relevant overloads outside its admitted scope.

Start with the adapted ScalarFunctionsTest SQL and runtime-source SQL parity tests, then compare the four listed STRING function groups with native Calc execution and Flink fallback. Done means verified values, resolved types, NULL and boundary behavior, safe fallback for unverified forms, and the admitted scope documented in docs/operators/calc-filter.md.

Written by the indexing model from the issue text.

Native STRING helpers: track FROM_BASE64, numeric SPLIT_INDEX, dynamic SHA2 and classification gaps

Description

Contributor guide

First steps

Research direction

Assessment