apache / apache/gluten

free(): invalid pointer in Reading SequenceFileInputFormat

Open
#6,897 2 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

Enviroment
- debian-10.13
- Spark-3.2.2
- gluten-1.2.0 rc2
- jdk: openjdk-8

When execute the sql below, the job will fail and throw message `free(): invalid pointer`
```sql
select count(udid) from table where ds = '20240731';
```

The gluten plan is :
```
== Fallback Summary ==
(1) Scan hive xxxtable: Unsupported file format for UnknownFormat.

== Physical Plan ==
VeloxColumnarToRowExec (7)
+- ^ ProjectExecTransformer (5)
+- ^ InputIteratorTransformer (4)
+- RowToVeloxColumnar (2)
+- Scan hive xxxtable (1)
```

The table fileformat is SequenceFileInputFormat
```
# Storage Information
SerDe Library: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
InputFormat: org.apache.hadoop.mapred.SequenceFileInputFormat
OutputFormat: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
```

The error message is:

```
[2024-08-16 17:58:22.422]Container exited with a non-zero exit code 134. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
/bin/bash:行 1: 23131 已放弃 /home/hadoop/java-8-sun/bin/java -server -Xmx4096m
Last 4096 bytes of stderr :

[spark.gluten.sql.session.timeZone.default, Asia/Hong_Kong]
[spark.sql.caseSensitive, false]
I20240816 17:58:22.259377 82482 FileSystems.cpp:161] LocalFileSystem::mkdir /disk4/yarn/local/usercache/hadoop/appcache/xxx
I20240816 17:58:22.321003 82482 Task.cpp:1136] All drivers (1) finished for task Gluten_Stage_3_TID_12_VTID_0 after running for 51 ms.
I20240816 17:58:22.321048 82482 Task.cpp:1876] Terminating task Gluten_Stage_3_TID_12_VTID_0 with state Finished after running for 51 ms.
I20240816 17:58:22.321477 82482 WholeStageResultIterator.cc:319] Native Plan with stats for: [Stage: 3 TID: 12]
-- Aggregation[1][PARTIAL n1_0 := count_partial("n0_0")] -> n1_0:BIGINT
Output: 1 rows (32B, 1 batches), Cpu time: 133.99us, Blocked wall time: 0ns, Peak memory: 64.25KB, Memory allocations: 3, Threads: 1
hashtable.capacity sum: 0, count: 1, min: 0, max: 0
hashtable.numDistinct sum: 0, count: 1, min: 0, max: 0
hashtable.numRehashes sum: 0, count: 1, min: 0, max: 0
hashtable.numTombstones sum: 0, count: 1, min: 0, max: 0
queuedWallNanos sum: 7.00us, count: 1, min: 7.00us, max: 7.00us
runningAddInputWallNanos sum: 99.58us, count: 1, min: 99.58us, max: 99.58us
runningFinishWallNanos sum: 0ns, count: 1, min: 0ns, max: 0ns
runningGetOutputWallNanos sum: 30.72us, count: 1, min: 30.72us, max: 30.72us
-- ValueStream[0][] -> n0_0:VARCHAR
Input: 0 rows (0B, 0 batches), Output: 1271 rows (71.81KB, 1 batches), Cpu time: 48.92ms, Blocked wall time: 0ns, Peak memory: 0B, Memory allocations: 0, Threads: 1
runningAddInputWallNanos sum: 0ns, count: 1, min: 0ns, max: 0ns
runningFinishWallNanos sum: 9.92us, count: 1, min: 9.92us, max: 9.92us
runningGetOutputWallNanos sum: 49.98ms, count: 1, min: 49.98ms, max: 49.98ms

free(): invalid pointer
.
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2455)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2404)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2403)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2403)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1160)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1160)
at scala.Option.foreach(Option.scala:407)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1160)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2643)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2585)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2574)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)

at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:70)
at org.apache.kyuubi.operation.ExecuteStatement.waitStatementComplete(ExecuteStatement.scala:135)
at org.apache.kyuubi.operation.ExecuteStatement.$anonfun$runInternal$1(ExecuteStatement.scala:173)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748) (state=,code=0)
```

We have set `spark.gluten.sql.debug=true`, but it seems that the only useful information is `free(): invalid pointer`.

I guess there is something wrong is `RowToVeloxColumnar`.

### Spark version

Spark-3.2.x

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.