apache / apache/gluten

unable to run Gluten compiled with AOCC derived from clang16

Open
#5,831 8 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Backend

VL (Velox)

### Bug description

```
zetta_spark3@dc78cde26b0c:~$ ./spark/bin/spark-sql --master local --conf spark.executor.instances=3 --conf spark.executor.cores=5 --conf spark.memory.offHeap.enabled=true --conf spark.executor.memory=40g --conf spark.driver.cores=5 --driver-memory 50g --conf spark.plugins=io.glutenproject.GlutenPlugin --conf spark.memory.offHeap.enabled=true --conf spark.memory.offHeap.size=20g --conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager --conf spark.gluten.sql.columnar.backend.velox.SplitPreloadPerDriver=0 --database tpcds_100g -f tpcds-data-gen/spark-sql-perf/src/main/resources/tpcds_2_4/q3.sql
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
24/05/17 15:02:14 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
24/05/17 15:02:17 WARN HiveConf: HiveConf of name hive.stats.jdbc.timeout does not exist
24/05/17 15:02:17 WARN HiveConf: HiveConf of name hive.stats.retries.wait does not exist
24/05/17 15:02:19 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 2.3.0
24/05/17 15:02:19 WARN ObjectStore: setMetaStoreSchemaVersion called but recording version is disabled: version = 2.3.0, comment = Set by MetaStore zetta_spark3@172.17.0.2
24/05/17 15:02:19 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
Spark master: local, Application Id: local-1715938336665
24/05/17 15:02:22 WARN SparkShimProvider: Spark runtime version 3.3.2 is not matched with Gluten's fully tested version 3.3.1
E0517 15:02:23.888530 1561776 Exceptions.h:69] Line: /home/zetta/incubator-gluten-1.1.1/ep/build-velox/build/velox_ep/./velox/common/memory/MemoryPool.h:881, Function:sanityCheckLocked, Expression: Bad memory usage track state: Memory Pool[op.0.0.0.TableScan LEAF root[WholeStageIterator_root] parent[node.0] MALLOC track-usage thread-safe], Source: RUNTIME, ErrorCode: INVALID_STATE
terminate called after throwing an instance of 'facebook::velox::VeloxRuntimeError'
what(): Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Bad memory usage track state: Memory Pool[op.0.0.0.TableScan LEAF root[WholeStageIterator_root] parent[node.0] MALLOC track-usage thread-safe]
Retriable: False
Function: sanityCheckLocked
File: /home/zetta/incubator-gluten-1.1.1/ep/build-velox/build/velox_ep/./velox/common/memory/MemoryPool.h
Line: 881
Stack trace:
# 0 facebook::velox::VeloxException::VeloxException(char const*, unsigned long, char const*, std::basic_string_view >, std::basic_string_view >, std::basic_string_view >, std::basic_string_view >, bool, facebook::velox::VeloxException::Type, std::basic_string_view >)
# 1 void facebook::velox::detail::veloxCheckFail, std::allocator > const&>(facebook::velox::detail::VeloxCheckFailArgs const&, std::__cxx11::basic_string, std::allocator > const&)
# 2 facebook::velox::memory::MemoryPoolImpl::releaseThreadSafe(unsigned long, bool)
# 3 facebook::velox::FlatVector::~FlatVector()
# 4 facebook::velox::DictionaryVector::~DictionaryVector()
# 5 facebook::velox::RowVector::~RowVector()
# 6 void __gnu_cxx::new_allocator::destroy(gluten::VeloxColumnarBatch*)
# 7 std::_Hashtable >, std::allocator > >, std::__detail::_Select1st, std::equal_to, std::hash, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits >::_M_erase(unsigned long, std::__detail::_Hash_node_base*, std::__detail::_Hash_node >, false>*)
# 8 gluten::ObjectStore::release(long)
# 9 Java_io_glutenproject_columnarbatch_ColumnarBatchJniWrapper_close
# 10 0x00007ff1ad017da6
# 11 0x00007ff1ad007fd3
# 12 0x00007ff1ad007fd3
# 13 0x00007ff1ad007fd3
# 14 0x00007ff1ad007fd3
# 15 0x00007ff1ad007fd3
# 16 0x00007ff1ad007d5f
# 17 0x00007ff1aeff030f
```

### Spark version

Spark-3.3.x

### Spark configurations

Spark config:
(spark.app.name,org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver)
(spark.app.submitTime,1716290104384)
(spark.driver.cores,5)
(spark.driver.memory,50g)
(spark.eventLog.dir,file:///home/zetta_spark3/sparkEventLogs)
(spark.eventLog.enabled,true)
(spark.executor.cores,5)
(spark.executor.instances,3)
(spark.executor.memory,40g)
(spark.executor.memoryOverhead,500m)
(spark.hadoop.spark.parquet.binaryAsString,false)
(spark.hadoop.spark.sql.caseSensitive,false)
(spark.history.fs.logDirectory,file:///home/zetta_spark3/sparkEventLogs)
(spark.jars,)
(spark.kryoserializer.buffer.max,1g)
(spark.master,local)
(spark.memory.offHeap.enabled,true)
(spark.memory.offHeap.size,20g)
(spark.plugins,io.glutenproject.GlutenPlugin)
(spark.repl.local.jars,)
(spark.serializer,org.apache.spark.serializer.KryoSerializer)
(spark.shuffle.manager,org.apache.spark.shuffle.sort.ColumnarShuffleManager)
(spark.sql.adaptive.advisoryPartitionSizeInBytes,100M)
(spark.sql.autoBroadcastJoinThreshold,200M)
(spark.sql.catalogImplementation,hive)
(spark.sql.dynamicPartitionPruning.enabled,true)
(spark.sql.join.preferSortMergeJoin,false)
(spark.sql.legacy.parquet.nanosAsLong,true)
(spark.sql.optimizer.dynamicPartitionPruning.enforceBroadcastReuse,true)
(spark.sql.shuffle.partitions,400)
(spark.submit.deployMode,client)
(spark.submit.pyFiles,)
(spark.ui.retainedJobs,5000)
(spark.ui.retainedStages,5000)
(spark.ui.showConsoleProgress,true)

### System information

System information
zetta@dc78cde26b0c:~$ bash incubator-gluten-1.1.1/dev/info.sh

Velox System Info v0.0.2
Commit: Not in a git repo.
CMake Version: 3.22.1
System: Linux-5.15.0-107-generic
Arch: x86_64
C++ Compiler: /opt/AMD/aocc-compiler-4.2.0/bin/clang++
C++ Compiler Version: 16.0.3
C Compiler: /opt/AMD/aocc-compiler-4.2.0/bin/clang
C Compiler Version: 16.0.3
CMake Prefix Path: /usr/local;/usr;/;/usr;/usr/local;/usr/X11R6;/usr/pkg;/opt

### Relevant logs

_No response_

Contributor guide

Open the contributing guide

Research direction

Run the reported spark-sql command with the Velox backend and AOCC compiler, then inspect the failure at velox/common/memory/MemoryPool.h:881. Use dev/info.sh to confirm the environment and compare the memory-pool accounting around the stack shown. Done means the query completes without the VeloxRuntimeError and memory tracking remains consistent.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, linux
Domain
backend, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.