apache / apache/gluten

[VL] FileNotFound libarrow_cdata_jni.so on reduce operation like count on Spark 3.5

Open
#6,996 1 comment 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

Trying to read from s3 path like spark.read.format("parquet").load()

Getting exception as below
### Bug description

Exception Trace

24[/08/23](https://darwin.dream11.com/08/23) 05:32:46 ERROR TaskResources: Task 17 failed by error:
java.lang.IllegalStateException: error loading native libraries: java.io.FileNotFoundException: x86_64[/libarrow_cdata_jni.so](https://darwin.dream11.com/libarrow_cdata_jni.so)
at org.apache.arrow.c.jni.JniLoader.load(JniLoader.java:95)
at org.apache.arrow.c.jni.JniLoader.loadRemaining(JniLoader.java:76)
at org.apache.arrow.c.jni.JniLoader.ensureLoaded(JniLoader.java:60)
at org.apache.arrow.c.jni.JniWrapper.get(JniWrapper.java:27)
at org.apache.arrow.c.SchemaExporter.export(SchemaExporter.java:117)
at org.apache.arrow.c.Data.exportField(Data.java:59)
at org.apache.arrow.c.Data.exportSchema(Data.java:75)
at org.apache.gluten.utils.ArrowAbiUtil$.exportSchema(ArrowAbiUtil.scala:134)
at org.apache.gluten.execution.RowToVeloxColumnarExec$.toColumnarBatchIterator(RowToVeloxColumnarExec.scala:125)
at org.apache.gluten.execution.RowToVeloxColumnarExec.$anonfun$doExecuteColumnarInternal$2(RowToVeloxColumnarExec.scala:72)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2(RDD.scala:855)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2$adapted(RDD.scala:855)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)
at org.apache.gluten.execution.ColumnarInputRDDsWrapper.$anonfun$getIterators$1(WholeStageTransformer.scala:445)
at scala.collection.immutable.List.flatMap(List.scala:366)
at org.apache.gluten.execution.ColumnarInputRDDsWrapper.getIterators(WholeStageTransformer.scala:436)
at org.apache.gluten.execution.WholeStageZippedPartitionsRDD.$anonfun$compute$1(WholeStageZippedPartitionsRDD.scala:48)
at org.apache.gluten.utils.Arm$.withResource(Arm.scala:25)
at org.apache.gluten.metrics.GlutenTimeMetric$.millis(GlutenTimeMetric.scala:37)
at org.apache.gluten.execution.WholeStageZippedPartitionsRDD.compute(WholeStageZippedPartitionsRDD.scala:46)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
at org.apache.spark.scheduler.Task.run(Task.scala:141)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
at java.base[/java.util.concurrent.ThreadPoolExecutor.runWorker](https://darwin.dream11.com/java.util.concurrent.ThreadPoolExecutor.runWorker)(ThreadPoolExecutor.java:1128)
at java.base[/java.util.concurrent.ThreadPoolExecutor](https://darwin.dream11.com/java.util.concurrent.ThreadPoolExecutor)$Worker.run(ThreadPoolExecutor.java:628)
at java.base[/java.lang.Thread.run](https://darwin.dream11.com/java.lang.Thread.run)(Thread.java:829)

### Spark version

Spark-3.5.x

### Spark configurations

from pyspark.sql import SparkSession
spark = SparkSession.builder \
.appName("Gluten-Velox Example") \
.config("spark.plugins", "org.apache.gluten.GlutenPlugin") \
.config("spark.jars", "/home/ray/.ivy2/jars/com.amazonaws_aws-java-sdk-bundle-1.12.262.jar,/home/ray/.ivy2/jars/org.apache.spark_spark-hadoop-cloud_2.12-3.5.0.jar,/home/ray/.ivy2/jars/org.apache.hadoop_hadoop-aws-3.3.4.jar,https://d11-icebergtable.s3.amazonaws.com/velox/velox-latest/gluten-velox-bundle-spark3.5_2.12-ubuntu_20.04_x86_64-1.2.0.jar,https://d11-icebergtable.s3.amazonaws.com/velox/velox-latest/gluten-thirdparty-lib-ubuntu-20.04-x86_64.jar") \
.config("spark.gluten.loadLibFromJar", "true") \
.config("spark.memory.offHeap.enabled","true") \
.config("spark.memory.offHeap.size","1G") \
.config("spark.hadoop.fs.s3a.secret.key", "MGUBII62jZDNYFIpu117oP/MyON5WOEzq3KALdJr") \
.config("spark.hadoop.fs.s3a.access.key","AKIARENNYYZKYIMINT23") \
.config("spark.hadoop.fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem") \
.config("spark.hadoop.fs.s3a.aws.credentials.provider","org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider") \
.config("spark.hadoop.fs.s3a.path.style.access","false") \
.config("spark.hadoop.fs.s3a.endpoint","https://s3.us-east-1.amazonaws.com") \
.config("spark.jars.packages","com.amazonaws:aws-java-sdk-bundle:1.12.262,org.apache.hadoop:hadoop-aws:3.3.4,org.apache.spark:spark-hadoop-cloud_2.12:3.5.0") \
.getOrCreate()

### System information

Arch : x86_64
CPU : Intel Xeon
OS: Ubuntu

### Relevant logs

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the Spark 3.5 count or reduce operation with the shown Velox configuration and inspect the native-loading path through org.apache.arrow.c.jni.JniLoader and org.apache.gluten.execution.RowToVeloxColumnarExec. Check how spark.gluten.loadLibFromJar handles libarrow_cdata_jni.so; done means the operation runs without the FileNotFoundException on the stated Ubuntu x86_64 setup.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java, scala, ubuntu
Domain
backend, data, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.