[VL] Spark execute muilti tpcds sql (tpcds tt test mode), Gluten report error or crash

### Backend

VL (Velox)

### Bug description

[Expected behavior] and [actual behavior].
spark vanilla execute tpcds tt ( executor 4 tpcds 99 queries same time)test mode pass and no error report;

### Spark version

None

### Spark configurations

spark.master yarn
spark.deploy-mode client
spark.eventLog.enabled true
spark.eventLog.dir hdfs://master:9000/sparklogs
spark.driver.cores 8
spark.driver.memory 9g
spark.driver.maxResultSize 27g
spark.executor.instances 24
spark.executor.cores 4
spark.executor.memory 6g
spark.executor.memoryOverhead 1g
spark.memory.offHeap.enabled true
spark.gluten.enabled true
spark.plugins io.glutenproject.GlutenPlugin
spark.gluten.sql.columnar.backend.lib velox
spark.shuffle.manager org.apache.spark.shuffle.sort.ColumnarShuffleManager
spark.executorEnv.VELOX_HDFS hdfs://master:9000
spark.gluten.loadLibFromJar true
spark.io.compression.codec lz4
spark.memory.offHeap.size 10g
spark.task.maxFailures 1
spark.default.parallelism 192
spark.kryoserializer.buffer 288k
spark.memory.fraction 0.55
spark.shuffle.file.buffer 128k
spark.sql.adaptive.maxShuffledHashJoinLocalMapThreshold 8m
spark.sql.autoBroadcastJoinThreshold 1m
spark.sql.broadcastTimeout 6900
spark.sql.files.maxPartitionBytes 384m
spark.sql.files.openCostInBytes 3m
spark.sql.shuffle.partitions 192

### System information

_No response_

### Relevant logs

```bash
error1：

Error: org.apache.hive.service.cli.HiveSQLException: Error running query: org.apache.spark.SparkException: Job aborted due to stage failure: Task 134 in stage 27290.0 failed 1 times, most recent failure: Lost task 134.0 in stage 27290.0 (TID 1215849) (node3 executor 8): ExecutorLostFailure (executor 8 exited caused by one of the running tasks) Reason: Container from a bad node: container_1702888231711_0041_01_000009 on host: node3. Exit status: 134. Diagnostics: 041_01_000009/stderr
Last 4096 bytes of stderr :
utor$Worker.run(ThreadPoolExecutor.java:624)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 more
, Source: RUNTIME, ErrorCode: INVALID_STATE
E1226 12:51:21.793781 213655 Exceptions.h:69] Line: ../../velox/exec/Driver.cpp:550, Function:runInternal, Expression: Operator::getOutput failed for [operator: ValueStream, plan node ID: 1]: Error during calling Java code from native code: org.apache.spark.shuffle.FetchFailedException
at org.apache.spark.errors.SparkCoreErrors$.fetchFailedError(SparkCoreErrors.scala:312)
at org.apache.spark.storage.ShuffleBlockFetcherIterator.throwFetchFailedException(ShuffleBlockFetcherIterator.scala:1180)
at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:918)
at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:85)
at org.apache.spark.util.CompletionIterator.next(CompletionIterator.scala:29)
at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:486)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:492)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:31)
at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at scala.collection.convert.Wrappers$IteratorWrapper.hasNext(Wrappers.scala:32)
at io.glutenproject.vectorized.GeneralInIterator.hasNext(GeneralInIterator.java:31)
at io.glutenproject.vectorized.ColumnarBatchOutIterator.nativeHasNext(Native Method)
at io.glutenproject.vectorized.ColumnarBatchOutIterator.hasNextInternal(ColumnarBatchOutIterator.java:65)
at io.glutenproject.vectorized.GeneralOutIterator.hasNext(GeneralOutIterator.java:37)
at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:45)
at io.glutenproject.utils.IteratorCompleter.hasNext(Iterators.scala:69)
at io.glutenproject.utils.PayloadCloser.hasNext(Iterators.scala:35)
at io.glutenproject.utils.PipelineTimeAccumulator.hasNext(Iterators.scala:98)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at org.apache.spark.shuffle.ColumnarShuffleWriter.internalWrite(ColumnarShuffleWriter.scala:102)
at org.apache.spark.shuffle.ColumnarShuffleWriter.write(ColumnarShuffleWriter.scala:218)
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
at org.apache.spark.scheduler.Task.run(Task.scala:136)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.spark.ExecutorDeadException: The relative remote executor(Id: 22), which maintains the block data to fetch is dead.
at org.apache.spark.network.netty.NettyBlockTransferService$$anon$2.createAndStart(NettyBlockTransferService.scala:136)
at org.apache.spark.network.shuffle.RetryingBlockTransferor.transferAllOutstanding(RetryingBlockTransferor.java:173)
at org.apache.spark.network.shuffle.RetryingBlockTransferor.lambda$initiateRetry$0(RetryingBlockTransferor.java:206)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 more
, Source: RUNTIME, ErrorCode: INVALID_STATE

error2:

/bin/bash: line 1: 213071 Aborted (core dumped) /usr/lib/jvm/java/bin/java -server -Xmx4096m '-XX:+IgnoreUnrecognizedVMOptions' '--add-opens=java.base/java.lang=ALL-UNNAMED' '--add-opens=java.base/java.lang.invoke=ALL-UNNAMED' '--add-opens=java.base/java.lang.reflect=ALL-UNNAMED' '--add-opens=java.base/java.io=ALL-UNNAMED' '--add-opens=java.base/java.net=ALL-UNNAMED' '--add-opens=java.base/java.nio=ALL-UNNAMED' '--add-opens=java.base/java.util=ALL-UNNAMED' '--add-opens=java.base/java.util.concurrent=ALL-UNNAMED' '--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED' '--add-opens=java.base/sun.nio.ch=ALL-UNNAMED' '--add-opens=java.base/sun.nio.cs=ALL-UNNAMED' '--add-opens=java.base/sun.security.action=ALL-UNNAMED' '--add-opens=java.base/sun.util.calendar=ALL-UNNAMED' '--add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED' -Djava.io.tmpdir=/data/data5/local/usercache/root/appcache/application_1702888231711_0041/container_1702888231711_0041_01_000009/tmp '-Dspark.driver.port=35929' -Dspark.yarn.app.container.log.dir=/data/data5/log/application_1702888231711_0041/container_1702888231711_0041_01_000009 -XX:OnOutOfMemoryError='kill %p' org.apache.spark.executor.YarnCoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@master:35929 --executor-id 8 --hostname node3 --cores 4 --app-id application_1702888231711_0041 --resourceProfileId 0 > /data/data5/log/application_1702888231711_0041/container_1702888231711_0041_01_000009/stdout 2> /data/data5/log/application_1702888231711_0041/container_1702888231711_0041_01_000009/stderr
Last 4096 bytes of stderr :
utor$Worker.run(ThreadPoolExecutor.java:624)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 more
, Source: RUNTIME, ErrorCode: INVALID_STATE
E1226 12:51:21.793781 213655 Exceptions.h:69] Line: ../../velox/exec/Driver.cpp:550, Function:runInternal, Expression: Operator::getOutput failed for [operator: ValueStream, plan node ID: 1]: Error during calling Java code from native code: org.apache.spark.shuffle.FetchFailedException
at org.apache.spark.errors.SparkCoreErrors$.fetchFailedError(SparkCoreErrors.scala:312)
at org.apache.spark.storage.ShuffleBlockFetcherIterator.throwFetchFailedException(ShuffleBlockFetcherIterator.scala:1180)
at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:918)
at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:85)
at org.apache.spark.util.CompletionIterator.next(CompletionIterator.scala:29)
at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:486)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:492)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:31)
at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at scala.collection.convert.Wrappers$IteratorWrapper.hasNext(Wrappers.scala:32)
at io.glutenproject.vectorized.GeneralInIterator.hasNext(GeneralInIterator.java:31)
at io.glutenproject.vectorized.ColumnarBatchOutIterator.nativeHasNext(Native Method)
at io.glutenproject.vectorized.ColumnarBatchOutIterator.hasNextInternal(ColumnarBatchOutIterator.java:65)
at io.glutenproject.vectorized.GeneralOutIterator.hasNext(GeneralOutIterator.java:37)
at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:45)
at io.glutenproject.utils.IteratorCompleter.hasNext(Iterators.scala:69)
at io.glutenproject.utils.PayloadCloser.hasNext(Iterators.scala:35)
at io.glutenproject.utils.PipelineTimeAccumulator.hasNext(Iterators.scala:98)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at org.apache.spark.shuffle.ColumnarShuffleWriter.internalWrite(ColumnarShuffleWriter.scala:102)
at org.apache.spark.shuffle.ColumnarShuffleWriter.write(ColumnarShuffleWriter.scala:218)
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
at org.apache.spark.scheduler.Task.run(Task.scala:136)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.spark.ExecutorDeadException: The relative remote executor(Id: 22), which maintains the block data to fetch is dead.
at org.apache.spark.network.netty.NettyBlockTransferService$$anon$2.createAndStart(NettyBlockTransferService.scala:136)
at org.apache.spark.network.shuffle.RetryingBlockTransferor.transferAllOutstanding(RetryingBlockTransferor.java:173)
at org.apache.spark.network.shuffle.RetryingBlockTransferor.lambda$initiateRetry$0(RetryingBlockTransferor.java:206)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 more
, Source: RUNTIME, ErrorCode: INVALID_STATE
```