datafusion-contrib / datafusion-contrib/StreamFusion

Native SQL/JSON: extend selectors beyond the verified definite-path grammar

Open
#91 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
120
Forks
8
Avg merge
7h 29m
Merged PRs (30d)
76

Description

Remaining work

Extend SQL/JSON path coverage beyond the verified definite member/index grammar. The definite-path whitespace and Unicode/quoted-member extensions have shipped on main, including PR #128. Empty quoted member names, including nested member/index paths, shipped in PR #131 (commit 48fb9ac89649dc6620b9ac325985f61cc82d070d). Negative literal array indexes also shipped in PR #150, preserving Flink's indexing, validation and error behavior. Standard JSON/Unicode escapes and printable ASCII escapes in quoted member names shipped in PR #156, with released-Flink parity and native execution checks. Escapes of literal Unicode and control characters shipped in PR #160. Trailing U+0000–U+0020 controls after an array index and before its closing bracket shipped in PR #163, with released-Flink parity and native execution metrics. Literal unquoted dot-member names, including punctuation, controls and well-formed Unicode, shipped in PR #172 (on main through PR #170). Wildcard selectors, including repeated wildcards and member/index continuations, shipped in PR #173 (286c5b88), with released-Flink parity, native runtime metrics and all required CI passing. Current coverage and exact fallback rules are documented in docs/operators/calc-filter.md.

Array-index unions ($[0,2,-1], including duplicate indexes and member/index/wildcard continuations) shipped in PR #175 (7b581539), after every required CI and unchanged upstream suite passed. Local validation passed 628 Java JSON/expression tests, 555 Rust engine/bridge tests and the unchanged upstream Calc/JSON suite (708 passed, two upstream skips), with actual native execution assertions.

Remaining coverage boundaries:

  • Non-Calc expression contexts, such as residual join predicates, retain the narrower native/fused-expression admission. General JSON path evaluation there is not yet covered by the whole-Calc bridge.
  • MAP and MULTISET boundaries, nested unsupported leaves, and non-Calc expression contexts remain outside shipped bridge admission. The MAP/MULTISET prototype in PR #202 is held as a draft because its measured workloads are slower.

Within the admitted scalar Calc boundary, PR #176 (00b8a172) ships released-Flink generation for recursive descent, filters, slices, member unions, path functions, additional whitespace/invalid paths, JSON_QUERY and dynamic JSON_VALUE/JSON_QUERY paths. The native definite/wildcard/array-union fast paths are retained. Dynamic JSON_EXISTS remains a released-Flink planning failure, not a missing StreamFusion implementation.

The original audit included nine instantiated literal-path rejections and one unexpanded parameterized record. Do not count unexpanded variants as verified support. Portable parameter expansion and coverage accounting shipped with #107 in PR #150; the original independently produced corpus was unavailable. A separate combinatorial query was blocked by CROSS JOIN (#101), not demonstrated JSON expression rejection.

Acceptance

  • Extend the executed released-Flink path matrix for each newly admitted selector, including boundaries, missing/null/scalar/container values and multiple matches.
  • Preserve strict/lax, ON EMPTY/ON ERROR, typed RETURNING behavior, complete-document validation and independent selection state for multiple paths in one projection.
  • Assert actual native execution for verified forms and explicit fallback for the remaining forms.
  • Keep the operator documentation current in the implementation commit.

Surrogate identity in intermediate JSON STRING results (#81), including downstream scalar consumers, also shipped in PR #150. That issue is complete.

PR #176 merged after all 25 required CI checks passed, including every unchanged upstream suite. Full Java validation, the unchanged upstream Calc/JSON suites, native work counters and release comparisons are recorded in the PR and coverage documentation.

Nested ARRAY/ROW Calc boundaries shipped in PR #199 (9dc1d9e7) after all 26 CI checks passed, including every upstream suite and the optimized image smoke test. The bridge retains one JVM callback per Arrow batch and borrows nested input views only while that batch is alive. Focused result/changelog, 5,003-row multi-batch and ownership tests pass. Release measurements are recorded as coverage groundwork, not a standalone speedup. Issue #91 remains open for the boundaries above.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the remaining coverage boundaries and the current matrix in docs/operators/calc-filter.md. Run the Java JSON/expression tests and Rust engine/bridge tests mentioned in the issue, then extend the released-Flink path matrix for an admitted form. Done means the acceptance cases pass, native execution and explicit fallback are asserted, and the documentation is updated.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, rust, sql
Domain
data-engineering, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.