apache / apache/gluten

[VL] Failures encountered in using S3a filesystem with latest 1.1.1 released jar

Open
#5,963 4 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

[Expected behavior] - IT should honor s3a filesystem and be able to access files.
[actual behavior] - It fails with the below error :
```
Reason: No registered file system matched with file path 's3a://perf-data-chstest1/
```

I am using the jar released at - https://github.com/apache/incubator-gluten/releases/download/v1.1.1/gluten-velox-bundle-spark3.3_2.12-1.1.1.jar

### Spark version

Spark-3.3.x

### Spark configurations

```
"spark.hadoop.fs.s3a.endpoint": "s3.direct.jp-tok.cloud-object-storage.appdomain.cloud",
"spark.hadoop.fs.s3a.access.key": "XXX",
"spark.hadoop.fs.s3a.secret.key": "XXX",
"spark.plugins": "io.glutenproject.GlutenPlugin",
"spark.memory.offHeap.enabled": "true",
"spark.memory.offHeap.size": "20g",
"spark.shuffle.manager": "org.apache.spark.shuffle.sort.ColumnarShuffleManager"
```

### System information

_No response_

### Relevant logs

```bash
Reason: No registered file system matched with file path 's3a://perf-data-chstest1/rasika_data/parquet_db/customer/20230907_030906_00076_8uifi_60b91066-caf5-4356-abe1-b9fdab
c81a4a'
Retriable: False
Context: Split [Hive: s3a://perf-data-chstest1/rasika_data/parquet_db/customer/20230907_030906_00076_8uifi_60b91066-caf5-4356-abe1-b9fdabc81a4a 0 - 134217728] Task Gluten_St
age_1_TID_1
Top-Level Context: Same as context.
Function: getFileSystem
File: /root/src/oap-project/gluten/ep/build-velox/build/velox_ep/velox/common/file/FileSystems.cpp
Line: 61
Stack trace:
# 0 _ZN8facebook5velox7process10StackTraceC1Ei
# 1 _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2 _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKSsEEvRKNS1_18VeloxCheckFailArgsET0_
# 3 _ZN8facebook5velox11filesystems13getFileSystemESt17basic_string_viewIcSt11char_traitsIcEESt10shared_ptrIKNS0_6ConfigEE
# 4 _ZN8facebook5velox19FileHandleGeneratorclERKSs
# 5 _ZN8facebook5velox13CachedFactoryISsSt10shared_ptrINS0_10FileHandleEENS0_19FileHandleGeneratorEE8generateERKSs
# 6 _ZN8facebook5velox9connector4hive11SplitReader12prepareSplitESt10shared_ptrINS0_6common14MetadataFilterEERNS0_4dwio6common17RuntimeStatisticsE
# 7 _ZN8facebook5velox9connector4hive14HiveDataSource8addSplitESt10shared_ptrINS1_14ConnectorSplitEE
# 8 _ZN8facebook5velox4exec9TableScan9getOutputEv
# 9 _ZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEE
# 10 _ZN8facebook5velox4exec6Driver4nextERSt10shared_ptrINS1_13BlockingStateEE
# 11 _ZN8facebook5velox4exec4Task4nextEPN5folly10SemiFutureINS3_4UnitEEE
# 12 _ZN6gluten24WholeStageResultIterator4nextEv
# 13 Java_io_glutenproject_vectorized_ColumnarBatchOutIterator_nativeHasNext
# 14 ffi_call_unix64
# 15 ffi_call_int
# 16 _ZN32VM_BytecodeInterpreterCompressed3runEP10J9VMThread
# 17 bytecodeLoopCompressed
# 18 0x00000000001a3a72

at io.glutenproject.vectorized.GeneralOutIterator.hasNext(GeneralOutIterator.java:39)
at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:45)
at io.glutenproject.utils.InvocationFlowProtection.hasNext(Iterators.scala:135)
at io.glutenproject.utils.IteratorCompleter.hasNext(Iterators.scala:69)
at io.glutenproject.utils.PayloadCloser.hasNext(Iterators.scala:35)
at io.glutenproject.utils.PipelineTimeAccumulator.hasNext(Iterators.scala:98)
at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
at scala.collection.Iterator.isEmpty(Iterator.scala:387)
at scala.collection.Iterator.isEmpty$(Iterator.scala:387)
at org.apache.spark.InterruptibleIterator.isEmpty(InterruptibleIterator.scala:28)
at io.glutenproject.execution.VeloxColumnarToRowExec$.toRowIterator(VeloxColumnarToRowExec.scala:116)
at io.glutenproject.execution.VeloxColumnarToRowExec.$anonfun$doExecuteInternal$1(VeloxColumnarToRowExec.scala:80)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2(RDD.scala:855)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitions$2$adapted(RDD.scala:855)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:136)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:839)
Caused by: java.lang.RuntimeException: Exception: VeloxRuntimeError
```
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.