apache / apache/gluten

[VL] Velox cache feature broken: MemoryAllocator::allocateNonContiguous crush

Open
#4,622 3 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

I am using Glue + Velox, and the data is stored on S3. However, I found that a large portion of the task running time is spent on reading remote data. After discovering the local cache feature, I intend to enable it to verify its effectiveness. However, I encountered the following error:

`C [libvelox.so+0x602f34e] facebook::velox::memory::MemoryAllocator::allocateNonContiguous(unsigned long, facebook::velox::memory::Allocation&, std::function, unsigned long)+0x3e
`
I found the track log below, but it doesn't seem to show specific errors, it just runs to this line of code.

### Spark version

None

### Spark configurations

spark submit config:

$SPARK_HOME/bin/spark-submit \
--master $K8S_MASTER \
--deploy-mode cluster \
--name spark-tpch-gluten-1g-$i \
--conf spark.executor.instances=2 \
--conf spark.executor.memory=10G \
--conf spark.executor.cores=6 \
--conf spark.driver.cores=4 \
--conf spark.driver.memory=4g \
--conf spark.driver.maxResultSize=1g \
--conf spark.eventLog.enabled=true \
--conf spark.eventLog.dir=$EVENTLOG_DIR \
--conf spark.hadoop.fs.s3a.access.key=
--conf spark.hadoop.fs.s3a.secret.key=
--conf spark.hadoop.fs.s3a.endpoint=
--conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
--conf spark.hadoop.fs.s3a.use.instance.credentials=false \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
--conf spark.kubernetes.namespace=$NAMESPACE \
--conf spark.kubernetes.container.image=$IMAGE \
--conf spark.kubernetes.container.image.pullPolicy=Always \
--conf spark.kubernetes.driver.podTemplateFile=$DRIVER_TMPL \
--conf spark.kubernetes.executor.podTemplateFile=$EXEXUTOR_TMPL \
--conf spark.kubernetes.driver.name=spark-tpch-gluten \
--conf spark.sql.files.maxPartitionBytes=128m \
--conf spark.sql.shuffle.partitions=200 \
--conf spark.default.parallelism=200 \
--conf spark.sql.adaptive.enabled=true \
--conf spark.plugins=io.glutenproject.GlutenPlugin \
--conf spark.gluten.loadLibFromJar=true \
--conf spark.memory.offHeap.enabled=true \
--conf spark.memory.offHeap.size=6g \
--conf spark.executor.memoryOverhead=4g \
--conf spark.gluten.sql.debug=true \
--conf spark.gluten.sql.injectNativePlanStringToExplain=true \
--conf spark.gluten.ui.enabled=true \
--conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
--conf spark.gluten.sql.columnar.backend.velox.glogSeverityLevel=0 \
--conf spark.shuffle.service.enabled=false \
--conf spark.sql.adaptive.localShuffleReader.enabled=false \
--conf spark.gluten.sql.columnar.backend.velox.cacheEnabled=true \
--conf spark.gluten.sql.columnar.backend.velox.ssdCacheSize=0 \
--conf spark.gluten.sql.columnar.backend.velox.memCacheSize=1073741824 \

spark UI info:
Backend Velox
Backend Branch HEAD
Backend Revision 8d12a9b36d4f754c3194aa8612f62c0d5395d541
Backend Revision Time 2024-01-11 16:26:31 +0800
GCC Version gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0
Gluten Branch fix-finalized-s3
Gluten Build Time 2024-01-11T08:45:18Z
Gluten Repo URL http://code.oppoer.me/bdc/spark/gluten.git
Gluten Revision c03dfeda16e3784e1afc87fab8965c5cf0d618a8
Gluten Revision Time 2024-01-11 16:41:35 +0800
Gluten Version 1.1.0-SNAPSHOT
Hadoop Version 2.7.4
Java Version 11.0.21
Scala Version 2.12.15
Spark Version 3.4.1

### System information

Velox System Info v0.0.2
Commit: 58d80a1abcaeab5f174e40b4f74fe6b209700cad
CMake Version: 3.16.3
System:
Arch:
C++ Compiler:
C++ Compiler Version:
C Compiler:
C Compiler Version:
CMake Prefix Path:

### Relevant logs

```bash
task log:
2024-02-02T08:13:26.034099154Z stderr F I0202 08:13:26.034029 87 HiveDataSource.cpp:262] No need to check type lowercase modeDATE
2024-02-02T08:13:26.034130125Z stderr F I0202 08:13:26.034093 87 HiveDataSource.cpp:546] Adding split Hive: s3a://xxxx-warehouse/user/hive/warehouse/tpch_1g_orc.db/lineitem/000000_0 4194304 - 4194304
2024-02-02T08:13:26.091046188Z stderr F I0202 08:13:26.090955 87 FileHandle.cpp:50] Generating file handle for: s3a://xxx-warehouse/user/hive/warehouse/tpch_1g_orc.db/lineitem/000000_0 uuid: 1
2024-02-02T08:13:26.09115247Z stdout F #
2024-02-02T08:13:26.091159517Z stdout F # A fatal error has been detected by the Java Runtime Environment:
2024-02-02T08:13:26.09116376Z stdout F #
2024-02-02T08:13:26.091166667Z stdout F # SIGSEGV (0xb) at pc=0x00007fc66de3934e, pid=16, tid=0x00007fc6931fb700
2024-02-02T08:13:26.091169891Z stdout F #
2024-02-02T08:13:26.091173795Z stdout F # JRE version: OpenJDK Runtime Environment (8.0_382-b05) (build 1.8.0_382-8u382-ga-1~20.04.1-b05)
2024-02-02T08:13:26.091178655Z stdout F # Java VM: OpenJDK 64-Bit Server VM (25.382-b05 mixed mode linux-amd64 compressed oops)
2024-02-02T08:13:26.091182465Z stdout F # Problematic frame:
2024-02-02T08:13:26.094288097Z stdout F # C [libvelox.so+0x602f34e] facebook::velox::memory::MemoryAllocator::allocateNonContiguous(unsigned long, facebook::velox::memory::Allocation&, std::function, unsigned long)+0x3e
2024-02-02T08:13:26.094300886Z stdout F #
2024-02-02T08:13:26.094303681Z stdout F # Core dump written. Default location: /opt/spark/work-dir/core or core.16
2024-02-02T08:13:26.094305871Z stdout F #
2024-02-02T08:13:26.094334563Z stdout F # An error report file with more information is saved as:
2024-02-02T08:13:26.094339558Z stdout F # /opt/spark/work-dir/hs_err_pid16.log
2024-02-02T08:13:26.100866576Z stderr F I0202 08:13:26.100792 88 FileSystems.cpp:161] LocalFileSystem::mkdir /var/data/spark-5fcdd701-9856-4651-9609-9f5b0f40f99c/gluten-8bc13c56-c338-472f-aa13-4144218fd4ba/gluten-spill/1e3fb351-95e5-4e9d-a65e-49016b90b277

hs_err_pid.log print stack trace:
Stack: [0x00007fc6930fb000,0x00007fc6931fc000], sp=0x00007fc6931f8bf0, free space=1014k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [libvelox.so+0x602f34e] facebook::velox::memory::MemoryAllocator::allocateNonContiguous(unsigned long, facebook::velox::memory::Allocation&, std::function, unsigned long)+0x3e

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j io.glutenproject.vectorized.ColumnarBatchOutIterator.nativeHasNext(J)Z+0
j io.glutenproject.vectorized.ColumnarBatchOutIterator.hasNextInternal()Z+5
j io.glutenproject.vectorized.GeneralOutIterator.hasNext()Z+1
J 2051 C1 scala.collection.convert.Wrappers$JIteratorWrapper.hasNext()Z (10 bytes) @ 0x00007fc739663db4 [0x00007fc739663c80+0x134]
j io.glutenproject.utils.InvocationFlowProtection.hasNext()Z+59
j io.glutenproject.utils.IteratorCompleter.hasNext()Z+4
j io.glutenproject.utils.PayloadCloser.hasNext()Z+8
j io.glutenproject.utils.PipelineTimeAccumulator.hasNext()Z+4
j org.apache.spark.InterruptibleIterator.hasNext()Z+11
j scala.collection.Iterator$$anon$10.hasNext()Z+4
j org.apache.spark.shuffle.ColumnarShuffleWriter.internalWrite(Lscala/collection/Iterator;)V+1
j org.apache.spark.shuffle.ColumnarShuffleWriter.write(Lscala/collection/Iterator;)V+2
j org.apache.spark.shuffle.ShuffleWriteProcessor.write(Lorg/apache/spark/rdd/RDD;Lorg/apache/spark/ShuffleDependency;JLorg/apache/spark/TaskContext;Lorg/apache/spark/Partition;)Lorg/apache/spark/scheduler/MapStatus;+46
j org.apache.spark.scheduler.ShuffleMapTask.runTask(Lorg/apache/spark/TaskContext;)Lorg/apache/spark/scheduler/MapStatus;+189
j org.apache.spark.scheduler.ShuffleMapTask.runTask(Lorg/apache/spark/TaskContext;)Ljava/lang/Object;+2
j org.apache.spark.TaskContext.runTaskWithListeners(Lorg/apache/spark/scheduler/Task;)Ljava/lang/Object;+2
j org.apache.spark.scheduler.Task.run(JILorg/apache/spark/metrics/MetricsSystem;ILscala/collection/immutable/Map;Lscala/Option;)Ljava/lang/Object;+254
j org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Lorg/apache/spark/executor/Executor$TaskRunner;Lscala/runtime/BooleanRef;)Ljava/lang/Object;+43
j org.apache.spark.executor.Executor$TaskRunner$$Lambda$746.apply()Ljava/lang/Object;+8
j org.apache.spark.util.Utils$.tryWithSafeFinally(Lscala/Function0;Lscala/Function0;)Ljava/lang/Object;+4
j org.apache.spark.executor.Executor$TaskRunner.run()V+457
j java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+95
j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5
j java.lang.Thread.run()V+11
v ~StubRoutines::call_stub
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.