apache / apache/gluten

[VL] Some filter conditions in MiscOperatorSuite failed

Open
#11,269 0 comments 1 reaction 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

Reverted PR https://github.com/facebookincubator/velox/pull/15565 in https://github.com/apache/incubator-gluten/pull/11266
in_or in_not and not in in MiscOperatorSuite failed with this pr. The failure is under investigation.

### Gluten version

main branch

### Spark version

None

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

```bash
Error:

libc++abi: terminating due to uncaught exception of type facebook::velox::VeloxUserError: Exception: VeloxUserError
Error Source: USER
Error Code: UNSUPPORTED
Reason: Filter(MultiRange, deterministic, no nulls): testInt64Range() is not supported.
Retriable: False
Context: Split [Hive: file:///Users/rong/workspace/github/apache/gluten/backends-velox/target/scala-2.13/test-classes/tpch-data-parquet/lineitem/part-00000-6c374e0a-7d76-401b-8458-a8e31f8ab704-c000.snappy.parquet 0 - 1863237] Task Gluten_Stage_0_TID_0_VTID_0
Function: testInt64Range
File: /Users/rong/workspace/github/apache/gluten/ep/build-velox/build/velox_ep/./velox/type/Filter.h
Line: 241

call stack from microbenchmark:

(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
* frame #0: 0x0000000128d47eb8 libvelox.dylib`facebook::velox::common::Filter::testInt64Range(this=0x0000600003a06cd0, (null)=1, (null)=2000, (null)=false) const at Filter.h:241:5
frame #1: 0x00000001298eadf8 libvelox.dylib`facebook::velox::common::(anonymous namespace)::testIntFilter(filter=0x0000600003a06cd0, intStats=0x00006000006453b0, mayHaveNull=false) at ScanSpec.cpp:231:20
frame #2: 0x00000001298eaad4 libvelox.dylib`facebook::velox::common::testFilter(filter=0x0000600003a06cd0, stats=0x00006000006453b0, totalRows=60175, type=std::__1::shared_ptr::element_type @ 0x0000000134021a90 strong=1 weak=1) at ScanSpec.cpp:382:14
frame #3: 0x000000012a780170 libvelox.dylib`facebook::velox::parquet::ParquetData::rowGroupMatches(this=0x0000600001a46d80, rowGroupId=0, filter=0x0000600003a06cd0) at ParquetData.cpp:89:12
frame #4: 0x000000012a77fe74 libvelox.dylib`facebook::velox::parquet::ParquetData::filterRowGroups(this=0x0000600001a46d80, scanSpec=0x000000011df32620, (null)=0, writerContext=0x000000011df36e48, result=0x000000016fdf9f90) at ParquetData.cpp:56:33
frame #5: 0x0000000129907bf0 libvelox.dylib`facebook::velox::dwio::common::SelectiveColumnReader::filterRowGroups(this=0x000000011df375e0, rowGroupSize=0, context=0x000000011df36e48, result=0x000000016fdf9f90) const at SelectiveColumnReader.cpp:66:16
frame #6: 0x000000012a78a070 libvelox.dylib`facebook::velox::parquet::StructColumnReader::filterRowGroups(this=0x000000011df36ec0, rowGroupSize=0, context=0x000000011df36e48, result=0x000000016fdf9f90) const at StructColumnReader.cpp:254:12
frame #7: 0x000000012a388c70 libvelox.dylib`facebook::velox::parquet::ParquetRowReader::Impl::filterRowGroups(this=0x000000011df36bc0) at ParquetReader.cpp:1251:20
frame #8: 0x000000012a3b63e0 libvelox.dylib`facebook::velox::parquet::ParquetRowReader::Impl::Impl(this=0x000000011df36bc0, readerBase=std::__1::shared_ptr::element_type @ 0x000000011df32fb8 strong=2 weak=1, options=0x000000011df32bf0) at ParquetReader.cpp:1237:5
frame #9: 0x000000012a3b6000 libvelox.dylib`facebook::velox::parquet::ParquetRowReader::Impl::Impl(this=0x000000011df36bc0, readerBase=std::__1::shared_ptr::element_type @ 0x000000011df32fb8 strong=2 weak=1, options=0x000000011df32bf0) at ParquetReader.cpp:1200:31
frame #10: 0x000000012a388a98 libvelox.dylib`std::__1::__unique_if::__unique_single std::__1::make_unique[abi:ne180100] const&, facebook::velox::dwio::common::RowReaderOptions const&>(__args=std::__1::shared_ptr::element_type @ 0x000000011df32fb8 strong=2 weak=1, __args=0x000000011df32bf0) at unique_ptr.h:597:30
frame #11: 0x000000012a3889dc libvelox.dylib`facebook::velox::parquet::ParquetRowReader::ParquetRowReader(this=0x0000600003649010, readerBase=std::__1::shared_ptr::element_type @ 0x000000011df32fb8 strong=2 weak=1, options=0x000000011df32bf0) at ParquetReader.cpp:1412:11
frame #12: 0x000000012a388b80 libvelox.dylib`facebook::velox::parquet::ParquetRowReader::ParquetRowReader(this=0x0000600003649010, readerBase=std::__1::shared_ptr::element_type @ 0x000000011df32fb8 strong=2 weak=1, options=0x000000011df32bf0) at ParquetReader.cpp:1411:52
frame #13: 0x000000012a389af0 libvelox.dylib`std::__1::__unique_if::__unique_single std::__1::make_unique[abi:ne180100] const&, facebook::velox::dwio::common::RowReaderOptions const&>(__args=std::__1::shared_ptr::element_type @ 0x000000011df32fb8 strong=2 weak=1, __args=0x000000011df32bf0) at unique_ptr.h:597:30
frame #14: 0x000000012a389a9c libvelox.dylib`facebook::velox::parquet::ParquetReader::createRowReader(this=0x0000600003459b80, options=0x000000011df32bf0) const at ParquetReader.cpp:1471:10
frame #15: 0x0000000131a6c6d8 libvelox.dylib`facebook::velox::connector::hive::SplitReader::createRowReader(this=0x000000011df32a30, metadataFilter=nullptr, rowType=std::__1::shared_ptr::element_type @ 0x0000600001a46c18 strong=5 weak=1, rowSizeTrackingEnabled= Has Value=false ) at SplitReader.cpp:408:33
frame #16: 0x0000000131a6bca4 libvelox.dylib`facebook::velox::connector::hive::SplitReader::prepareSplit(this=0x000000011df32a30, metadataFilter=nullptr, runtimeStats=0x000000011e8d4ff0, fileReadOps=0x000000016fdfa588) at SplitReader.cpp:188:3
frame #17: 0x0000000131a48ddc libvelox.dylib`facebook::velox::connector::hive::HiveDataSource::addSplit(this=0x000000011e8d4e00, split=std::__1::shared_ptr::element_type @ 0x000000011df32168 strong=5 weak=1) at HiveDataSource.cpp:363:17
frame #18: 0x0000000131f356f4 libvelox.dylib`facebook::velox::exec::TableScan::getSplit(this=0x000000011e82d400) at TableScan.cpp:390:20
frame #19: 0x0000000131f33b30 libvelox.dylib`facebook::velox::exec::TableScan::getOutput(this=0x000000011e82d400) at TableScan.cpp:175:32
frame #20: 0x0000000131c28a88 libvelox.dylib`facebook::velox::exec::(anonymous namespace)::getOutput(op=0x000000011e82d400, result=nullptr) at Driver.cpp:450:16
frame #21: 0x0000000131c286d8 libvelox.dylib`facebook::velox::exec::Driver::runInternal(std::__1::shared_ptr&, std::__1::shared_ptr&, std::__1::shared_ptr&)::$_3::operator()(this=0x000000016fdfb588) const at Driver.cpp:578:15
frame #22: 0x0000000131c11520 libvelox.dylib`void facebook::velox::exec::Driver::withDeltaCpuWallTimer&, std::__1::shared_ptr&, std::__1::shared_ptr&)::$_3>(this=0x0000600000d44ee0, op=0x000000011e82d400, opTimingMember=0x00000000000000e8, opFunction=0x000000016fdfb588) at Driver.cpp:1197:3
frame #23: 0x0000000131c0f778 libvelox.dylib`facebook::velox::exec::Driver::runInternal(this=0x0000600000d44ee0, self=std::__1::shared_ptr::element_type @ 0x0000600000d44ee0 strong=4 weak=4, blockingState=nullptr, result=nullptr) at Driver.cpp:575:13
frame #24: 0x0000000131c0e9d8 libvelox.dylib`facebook::velox::exec::Driver::next(this=0x0000600000d44ee0, future=0x000000016fdfbad0, blockingOp=0x000000016fdfbac8, blockingReason=0x000000016fdfbac4) at Driver.cpp:284:21
frame #25: 0x0000000131f5575c libvelox.dylib`facebook::velox::exec::Task::next(this=0x000000011e939a00, future=0x000000016fdfbdd8) at Task.cpp:884:29
frame #26: 0x00000001283a0eac libvelox.dylib`gluten::WholeStageResultIterator::next(this=0x000000011df11200) at WholeStageResultIterator.cc:237:23
frame #27: 0x0000000100036da8 generic_benchmark`gluten::ResultIterator::getNext(this=0x0000600002162b18) at ResultIterator.h:83:22
frame #28: 0x00000001000267b4 generic_benchmark`gluten::ResultIterator::hasNext(this=0x0000600002162b18) at ResultIterator.h:45:5
frame #29: 0x0000000100024a04 generic_benchmark`$_1::operator()(this=0x000000011f08deb8, state=0x000000016fdfc988, planFile="/tmp/micro/plan_9_0_0.json", splitFiles=size=1, dataFiles=size=0, localDirs=size=1, runtimeFactory= Lambda in File GenericBenchmark.cc at Line 709, readerType=kNone) const at GenericBenchmark.cc:462:28
frame #30: 0x0000000100023fe0 generic_benchmark`benchmark::internal::Benchmark* benchmark::RegisterBenchmark<$_1&, std::__1::basic_string, std::__1::allocator>&, std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>>&, std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>>&, std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&, std::__1::function&, gluten::FileReaderType>(this=0x000000011f08deb8, st=0x000000016fdfc988)>&, gluten::FileReaderType&&)::'lambda'(benchmark::State&)::operator()(benchmark::State&) const at benchmark.h:1115:41
frame #31: 0x0000000100023b38 generic_benchmark`benchmark::internal::LambdaBenchmark, std::__1::allocator>&, std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>>&, std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>>&, std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&, std::__1::function&, gluten::FileReaderType>(char const*, $_1&, std::__1::basic_string, std::__1::allocator>&, std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>>&, std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>>&, std::__1::vector, std::__1::allocator>, std::__1::allocator, std::__1::allocator>>> const&, std::__1::function&, gluten::FileReaderType&&)::'lambda'(benchmark::State&)>::Run(this=0x000000011f08de00, st=0x000000016fdfc988) at benchmark.h:1074:52
frame #32: 0x0000000100064d0c generic_benchmark`benchmark::internal::BenchmarkInstance::Run(unsigned long long, int, benchmark::internal::ThreadTimer*, benchmark::internal::ThreadManager*, benchmark::internal::PerfCountersMeasurement*) const + 72
frame #33: 0x0000000100077dd0 generic_benchmark`benchmark::internal::(anonymous namespace)::RunInThread(benchmark::internal::BenchmarkInstance const*, unsigned long long, int, benchmark::internal::ThreadManager*, benchmark::internal::PerfCountersMeasurement*) + 76
frame #34: 0x00000001000779fc generic_benchmark`benchmark::internal::BenchmarkRunner::DoNIterations() + 680
frame #35: 0x0000000100078498 generic_benchmark`benchmark::internal::BenchmarkRunner::DoOneRepetition() + 140
frame #36: 0x000000010005f1a4 generic_benchmark`benchmark::RunSpecifiedBenchmarks(benchmark::BenchmarkReporter*, benchmark::BenchmarkReporter*) + 2704
frame #37: 0x000000010000b9fc generic_benchmark`main(argc=1, argv=0x000000016fdff1d0) at GenericBenchmark.cc:758:3
frame #38: 0x000000018f1beb98 dyld`start + 6076
```

Contributor guide

Open the contributing guide

Research direction

Start with the failing in_or, in_not, and not-in cases in MiscOperatorSuite, then trace the reported path through velox/type/Filter.h, ScanSpec.cpp, and ParquetData.cpp. Reproduce the failure with the VL backend and determine the change needed for these filter conditions to pass without the unsupported testInt64Range error.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.