apache / apache/datafusion-comet

Native scans cannot propagate JVM-side Spark accumulators

Open
#3,867 1 comment 0 reactions 0 assignees View on GitHub
spark sql tests test
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

## Description

When using native Parquet scans (`COMET_PARQUET_SCAN_IMPL=native_datafusion`), Spark accumulators registered on the JVM side are not incremented because the native reader operates entirely in Rust via DataFusion and does not call back into JVM accumulator APIs.

This does not affect query correctness or user-visible behavior. Filter pushdown works correctly in native scans. The issue is that several Spark SQL tests use JVM-side accumulators (e.g. `NumRowGroupsAcc`) as a test mechanism to verify filter pushdown, and that mechanism does not work across the JNI boundary.

## Affected Tests

The following Spark SQL tests are currently ignored with `IgnoreCometNativeScan` due to this limitation:

- `filter pushdown - StringPredicate` (ParquetFilterSuite) - uses `NumRowGroupsAcc` to verify string predicate pushdown
- `Filters should be pushed down for vectorized Parquet reader at row group level` (ParquetFilterSuite) - uses accumulator to verify row group level filtering
- `SPARK-34562: Bloom filter push down` (ParquetFilterSuite) - uses accumulator to verify bloom filter pushdown

These tests are skipped across all Spark versions (3.4, 3.5, 4.0).

## Possible Solutions

1. Implement accumulator propagation from native scans back to JVM
2. Add equivalent native-side metrics that can be queried from JVM after execution
3. Write alternative Comet-specific tests that verify the same pushdown behavior without relying on accumulators

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.