datafusion-contrib / datafusion-contrib/StreamFusion

CI: account for native and fallback routes across upstream SQL tests

Open
#168 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
120
Forks
8
Avg merge
7h 29m
Merged PRs (30d)
76

Description

Shipped in #205: execution-audit.json now retains every JUnit case, exact executed/contracted/unclassified denominators, validated operator-work counts and fallback reasons. Five additional unchanged Calc contracts pass (32 cases total, 8 required native records, 24 explicitly unclassified). Missing/stale evidence and process failures remain blocking. This artifact does not infer native coverage for uncontracted cases or match duplicate parameterized reports without an invocation ID.

Shipped in #206 (merge b94064cc, all 26 checks passed): six unchanged Table API UDF methods now have route contracts across both backend variants. The focused run passes all 12 invocations: six require native Calc work and six require the exact Calc: unsupported function/operator: AS fallback. The full Table API class passes 56 cases with one upstream skip; its other 44 executed cases remain explicitly unclassified. The alias-wrapper admission gap is still open under #110.

Remaining work: comprehensive per-invocation route collection; batch, scan-only, deliberately unmodified-plan and host-failure classification; shared invocation identities; and fail-closed completeness inside each declared supported audit scope. Keep expanding verified operator/type contracts without treating planner installation as execution.

Original audit context:

The upstream suite's green test total does not establish native coverage. At main aea6056352c402e0c7008f6f223c7574324d796a, native-execution.tsv covers ten Flink test methods plus three Delta methods; all other passing tests lack that per-invocation execution contract.

In runtime job https://github.com/datafusion-contrib/StreamFusion/actions/runs/35210877386/job/105167816104?pr=150, the summary reports 8,571 passed tests but only 28 contracted invocations (15 native, 13 expected fallback). Those counts are not an overall native percentage. Agent unit tests also print simulated evidence, so raw log matches must not be mixed with actual SQL invocations.

The agent intentionally retains stock Flink for batch and selected unmodified-plan fixtures. Ordinary planner fallback messages are not present in that console log, and NativeExecution ignores callbacks outside an active contracted method. The newer portable audit improves representative coverage but does not classify the entire upstream suite.

Acceptance:

  • Record test identity and parameter/configuration variant, actual native operator work, and precise fallback reasons for upstream SQL invocations.
  • Distinguish native/mixed/full fallback, batch host-only, deliberately unmodified plans, scan-only plans, skips and host failures.
  • Keep simulated agent-unit-test evidence out of SQL totals.
  • Retain complete machine-readable artifacts and fail on missing/unclassified evidence within each declared supported audit scope.
  • Expand verified operator/type coverage incrementally and report the exact denominator; do not count installation alone as operator execution.

This extends the representative harness delivered for #106/#107 and the operator execution sampling. PR #31's installation check does not provide this route accounting. Related audit umbrella: #110.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the existing execution-audit.json artifact and native-execution.tsv coverage, then trace how upstream SQL invocations are collected and classified. Done means complete per-invocation identities, native/fallback and host-only classifications, machine-readable evidence, and fail-closed results within each declared audit scope without counting installation as execution.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql
Domain
ci-cd, data-engineering, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.