apache / apache/datafusion-comet
Native scans cannot propagate JVM-side Spark accumulators
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
## Description
When using native Parquet scans (`COMET_PARQUET_SCAN_IMPL=native_datafusion`), Spark accumulators registered on the JVM side are not incremented because the native reader operates entirely in Rust via DataFusion and does not call back into JVM accumulator APIs.
This does not affect query correctness or user-visible behavior. Filter pushdown works correctly in native scans. The issue is that several Spark SQL tests use JVM-side accumulators (e.g. `NumRowGroupsAcc`) as a test mechanism to verify filter pushdown, and that mechanism does not work across the JNI boundary.
## Affected Tests
The following Spark SQL tests are currently ignored with `IgnoreCometNativeScan` due to this limitation:
- `filter pushdown - StringPredicate` (ParquetFilterSuite) - uses `NumRowGroupsAcc` to verify string predicate pushdown
- `Filters should be pushed down for vectorized Parquet reader at row group level` (ParquetFilterSuite) - uses accumulator to verify row group level filtering
- `SPARK-34562: Bloom filter push down` (ParquetFilterSuite) - uses accumulator to verify bloom filter pushdown
These tests are skipped across all Spark versions (3.4, 3.5, 4.0).
## Possible Solutions
1. Implement accumulator propagation from native scans back to JVM
2. Add equivalent native-side metrics that can be queried from JVM after execution
3. Write alternative Comet-specific tests that verify the same pushdown behavior without relying on accumulators
Contributor guide
Assessment
This issue has not been assessed yet.