[VL] `free(): invalid pointer`
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Backend
VL (Velox)
### Bug description
We have a hive table in `SequenceFile` format, and when we execute this sql `select count(1) from test.test_table where dt='20250401';
`, it will report a `free(): invalid pointer`.
After debugging the code, I realized that by changing one line of code in `/incubator-gluten/cpp/velox/compute/VeloxBackend.cc`, the problem was solved. We suspect that the same pointer of `configs_` has been freed repeatedly.
before the change:
```
if (backendConf_->get(kDebugModeEnabled, false)) {
LOG(INFO) << "VeloxBackend config:" << printConfig(backendConf_->rawConfigs());
}
```
after the change:
```
LOG(INFO) << "VeloxBackend config:" << printConfig(backendConf_->rawConfigs());
```
### Gluten version
Gluten-1.3
### Spark version
Spark-3.4.x
### Spark configurations
${SPARK_HOME}/bin/spark-sql --master yarn \
--queue xxx \
--deploy-mode client \
--executor-cores 4 \
--executor-memory 8g \
--conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
--conf spark.plugins=org.apache.gluten.GlutenPlugin \
--conf spark.executorEnv.JAVA_HOME=/apps/svr/jdk-17.0.12 \
--conf spark.executor.extraJavaOptions="-Dio.netty.tryReflectionSetAccessible=true" \
--conf spark.sql.autoBroadcastJoinThreshold=-1 \
--conf spark.sql.adaptive.autoBroadcastJoinThreshold=-1
### System information
Gluten Version: 1.3.0
Commit: c205d4c7a54abfc1b866c2041192320eccfaa380
CMake Version: 3.28.3
System: Linux-6.8.0-59-generic
Arch: x86_64
CPU Name: Model name: Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz
C++ Compiler: /opt/rh/devtoolset-11/root/usr/bin/c++
C++ Compiler Version: 11.2.1
C Compiler: /opt/rh/devtoolset-11/root/usr/bin/cc
C Compiler Version: 11.2.1
CMake Prefix Path: /usr/local;/usr;/;/usr/local/lib64/python3.6/site-packages/cmake/data;/usr/local;/usr/X11R6;/usr/pkg;/opt
### Relevant logs
```bash
25/05/07 15:56:25 INFO [dispatcher-Executor] ExecutorPluginContainer: Initialized executor component for plugin org.apache.gluten.GlutenPlugin.
25/05/07 15:56:31 INFO [dispatcher-Executor] YarnCoarseGrainedExecutorBackend: Got assigned task 0
25/05/07 15:56:31 INFO [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)] Executor: Running task 0.0 in stage 0.0 (TID 0)
25/05/07 15:56:32 INFO [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)] TorrentBroadcast: Started reading broadcast variable 1 with 1 pieces (estimated total size 16.0 MiB)
25/05/07 15:56:32 INFO [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)] TransportClientFactory: Successfully created connection to /10.241.x.xx:3881 after 2 ms (0 ms spent in bootstraps)
25/05/07 15:56:32 INFO [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)] MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 49.1 KiB, free 8.6 GiB)
25/05/07 15:56:32 INFO [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)] TorrentBroadcast: Reading broadcast variable 1 took 78 ms
25/05/07 15:56:32 INFO [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)] MemoryStore: Block broadcast_1 stored as values in memory (estimated size 183.1 KiB, free 8.6 GiB)
25/05/07 15:56:32 INFO [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)] HadoopRDD: Input split: hdfs://xxx/ction_relation/dt=20250401/part-00000-29cadb76-3c64-45a2-b12a-804d4c64a8db-c000:0+16600316
25/05/07 15:56:32 INFO [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)] TorrentBroadcast: Started reading broadcast variable 0 with 1 pieces (estimated total size 16.0 MiB)
25/05/07 15:56:32 INFO [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)] MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 33.7 KiB, free 8.6 GiB)
25/05/07 15:56:32 INFO [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)] TorrentBroadcast: Reading broadcast variable 0 took 15 ms
25/05/07 15:56:32 INFO [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)] MemoryStore: Block broadcast_0 stored as values in memory (estimated size 478.5 KiB, free 8.6 GiB)
25/05/07 15:56:33 WARN [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)] Client: Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error
25/05/07 15:56:33 INFO [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)] CodecPool: Got brand-new decompressor [.snappy]
25/05/07 15:56:33 INFO [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)] CodecPool: Got brand-new decompressor [.snappy]
25/05/07 15:56:33 INFO [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)] CodecPool: Got brand-new decompressor [.snappy]
25/05/07 15:56:33 INFO [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)] CodecPool: Got brand-new decompressor [.snappy]
25/05/07 15:56:33 INFO [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)] CodeGenerator: Code generated in 172.209446 ms
25/05/07 15:56:33 INFO [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)] BaseAllocator: Debug mode disabled. Enable with the VM option -Darrow.memory.debug.allocator=true.
25/05/07 15:56:33 INFO [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)] DefaultAllocationManagerOption: allocation manager type not specified, using netty as the default type
25/05/07 15:56:33 INFO [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)] CheckAllocator: Using DefaultAllocationManager at memory/DefaultAllocationManagerFactory.class
W20250507 15:56:34.111573 2366039 TimeZoneMap.cpp:87] Unable to load [America/Ciudad_Juarez] from local timezone database: America/Ciudad_Juarez not found in timezone database
free(): invalid pointer
```
Contributor guide
Assessment
This issue has not been assessed yet.