[VL] Pool is null when using s3 cpp sdk to write data into s3 external table

Open

#6,714 opened on Aug 5, 2024 7 comments 0 reactions 0 assignees View on GitHub

bug triage

Dominant language: Scala
Stars: 1.6k
Forks: 657
Avg merge: 2d 14h
Merged PRs (30d): 80

Description

### Backend

VL (Velox)

### Bug description

Based gluten 468000c43efe7d54910431c7268768c4cb2d0410, when I test with velox to write data into s3 external table, I get the error as following:
```plain
ression: pool != nullptr , Source: RUNTIME, ErrorCode: INVALID_STATE
24/07/30 09:48:38 INFO CodecConfig: Compression: SNAPPY
E20240730 09:48:38.468577 93 Exceptions.h:67] Line: /gluten/ep/build-velox/build/velox_ep/velox/connectors/hive/storage_adapters/s3fs/S3FileSystem.cpp:229, Function:Impl, Expression: pool != nullptr , Source: RUNTIME, ErrorCode: INVALID_STATE
24/07/30 09:48:38 ERROR Utils: Aborting task
java.lang.RuntimeException: Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Retriable: False
Expression: pool != nullptr
Function: Impl
File: /gluten/ep/build-velox/build/velox_ep/velox/connectors/hive/storage_adapters/s3fs/S3FileSystem.cpp
Line: 229
Stack trace:
# 0 _ZN8facebook5velox7process10StackTraceC1Ei
# 1 _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2 _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorENS1_22CompileTimeEmptyStringEEEvRKNS1_18VeloxCheckFailArgsET0_
# 3 _ZN8facebook5velox11filesystems11S3WriteFile4ImplC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPNS0_21MultiRoleArnsS3ClientEPNS0_6memory10MemoryPoolE
# 4 _ZN8facebook5velox11filesystems11S3WriteFileC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPNS0_21MultiRoleArnsS3ClientEPNS0_6memory10MemoryPoolE
# 5 _ZN8facebook5velox11filesystems12S3FileSystem16openFileForWriteESt17basic_string_viewIcSt11char_traitsIcEERKNS1_11FileOptionsE
# 6 _ZN8facebook5velox11filesystems24s3WriteFileSinkGeneratorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS0_4dwio6common8FileSink7OptionsE
# 7 _ZNSt17_Function_handlerIFSt10unique_ptrIN8facebook5velox4dwio6common8FileSinkESt14default_deleteIS5_EERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS5_7OptionsEEPSK_E9_M_invokeERKSt9_Any_dataSG_SJ_
# 8 _ZN8facebook5velox4dwio6common8FileSink6createERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS3_7OptionsE
# 9 _ZN6gluten24VeloxParquetDatasourceS38initSinkERKSt13unordered_mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_St4hashIS7_ESt8equal_toIS7_ESaISt4pairIKS7_S7_EEE
# 10 _ZN6gluten22VeloxParquetDatasource4initERKSt13unordered_mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_St4hashIS7_ESt8equal_toIS7_ESaISt4pairIKS7_S7_EEE
# 11 Java_org_apache_gluten_datasource_DatasourceJniWrapper_nativeInitDatasource
# 12 0x00007ff08d018767
```
Exception stack:
```cpp
facebook::velox::filesystems::S3WriteFile::Impl::Impl(std::__cxx11::basic_string, std::allocator > const&, facebook::velox::MultiRoleArnsS3Client*, facebook::velox::memory::MemoryPool*)

facebook::velox::filesystems::S3WriteFile::S3WriteFile(std::__cxx11::basic_string, std::allocator > const&, facebook::velox::MultiRoleArnsS3Client*, facebook::velox::memory::MemoryPool*)

facebook::velox::filesystems::S3FileSystem::openFileForWrite(std::basic_string_view >, facebook::velox::filesystems::FileOptions const&)

facebook::velox::filesystems::s3WriteFileSinkGenerator(std::__cxx11::basic_string, std::allocator > const&, facebook::velox::dwio::common::FileSink::Options const&)

std::_Function_handler > (std::__cxx11::basic_string, std::allocator > const&, facebook::velox::dwio::common::FileSink::Options const&), std::unique_ptr > (*)(std::__cxx11::basic_string, std::allocator > const&, facebook::velox::dwio::common::FileSink::Options const&)>::_M_invoke(std::_Any_data const&, std::__cxx11::basic_string, std::allocator > const&, facebook::velox::dwio::common::FileSink::Options const&)

facebook::velox::dwio::common::FileSink::create(std::__cxx11::basic_string, std::allocator > const&, facebook::velox::dwio::common::FileSink::Options const&)

gluten::VeloxParquetDatasourceS3::initSink(std::unordered_map, std::allocator >, std::__cxx11::basic_string, std::allocator >, std::hash, std::allocator > >, std::equal_to, std::allocator > >, std::allocator, std::allocator > const, std::__cxx11::basic_string, std::allocator > > > > const&)

gluten::VeloxParquetDatasource::init(std::unordered_map, std::allocator >, std::__cxx11::basic_string, std::allocator >, std::hash, std::allocator > >, std::equal_to, std::allocator > >, std::allocator, std::allocator > const, std::__cxx11::basic_string, std::allocator > > > > const&)

org.apache.gluten.datasource.DatasourceJniWrapper#nativeInitDatasource
```

Spark sql is simple:
```sql
create table tbl (a INT, b INT) using parquet location 's3a:////'
insert overwrite table values(1, 2);
```

### Spark version

Spark-3.3.x

### Spark configurations

```
bin/spark-shell \
--packages org.apache.hadoop:hadoop-aws:3.3.1 \
--conf spark.plugins=org.apache.gluten.GlutenPlugin \
--conf spark.memory.offHeap.enabled=true \
--conf spark.memory.offHeap.size=20g \
--conf spark.driver.extraClassPath=/home/incubator-gluten/package/target/gluten-velox-bundle-spark3.3_2.12-centos_8_x86_64-1.2.0-SNAPSHOT.jar \
--conf spark.executor.extraClassPath=/home/incubator-gluten/package/target/gluten-velox-bundle-spark3.3_2.12-centos_8_x86_64-1.2.0-SNAPSHOT.jar \
--conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
--conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
--conf spark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider \
--conf spark.hadoop.fs.s3a.access.key=\
--conf spark.hadoop.fs.s3a.secret.key=\
--conf spark.hadoop.fs.s3a.endpoint=https://oss-ap-southeast-1.aliyuncs.com \
--conf spark.hadoop.fs.s3a.connection.ssl.enabled=true \
--conf spark.hadoop.fs.s3a.path.style.access=false \
--conf spark.eventLog.enabled=true \
--conf spark.gluten.velox.awsSdkLogLevel=DEBUG \
--conf spark.gluten.sql.native.writer.enabled=true
```

### System information

```
Velox System Info v0.0.2
Commit: 468000c43efe7d54910431c7268768c4cb2d0410
CMake Version: 3.28.3
System: Linux-4.18.0-348.7.1.el8_5.x86_64
Arch: x86_64
CPU Name: Model name: Intel(R) Xeon(R) Platinum 8369B CPU @ 2.70GHz
BIOS Model name: pc-i440fx-2.1
C++ Compiler: /opt/rh/gcc-toolset-9/root/usr/bin/c++
C++ Compiler Version: 9.2.1
C Compiler: /opt/rh/gcc-toolset-9/root/usr/bin/cc
C Compiler Version: 9.2.1
CMake Prefix Path: /usr/local;/usr;/;/usr/local/lib64/python3.9/site-packages/cmake/data;/usr/local;/usr/X11R6;/usr/pkg;/opt
```

### Relevant logs

```
spark.sql("insert overwrite db.tbl values(2, 1)")
24/08/22 09:58:58 WARN MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties
24/08/22 09:59:00 WARN AbstractS3ACommitterFactory: Using standard FileOutputCommitter to commit work. This is slow and potentially unsafe.
24/08/22 09:59:00 WARN AbstractS3ACommitterFactory: Using standard FileOutputCommitter to commit work. This is slow and potentially unsafe.
E20240822 09:59:06.508097 155687 Exceptions.h:67] Line: /home/incubator-gluten/ep/build-velox/build/velox_ep/velox/connectors/hive/storage_adapters/s3fs/S3FileSystem.cpp:227, Function:Impl, Expression: pool != nullptr , Source: RUNTIME, ErrorCode: INVALID_STATE
24/08/22 09:59:06 ERROR TaskResources: Task 3 failed by error:
java.lang.RuntimeException: Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Retriable: False
Expression: pool != nullptr
Function: Impl
File: /home/incubator-gluten/ep/build-velox/build/velox_ep/velox/connectors/hive/storage_adapters/s3fs/S3FileSystem.cpp
Line: 227
Stack trace:
# 0 _ZN8facebook5velox7process10StackTraceC1Ei
# 1 _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2 _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorENS1_22CompileTimeEmptyStringEEEvRKNS1_18VeloxCheckFailArgsET0_
# 3 _ZN8facebook5velox11filesystems11S3WriteFile4ImplC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPN3Aws2S38S3ClientEPNS0_6memory10MemoryPoolE
# 4 _ZN8facebook5velox11filesystems11S3WriteFileC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPN3Aws2S38S3ClientEPNS0_6memory10MemoryPoolE
# 5 _ZN8facebook5velox11filesystems12S3FileSystem16openFileForWriteESt17basic_string_viewIcSt11char_traitsIcEERKNS1_11FileOptionsE
# 6 _ZN8facebook5velox11filesystems24s3WriteFileSinkGeneratorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS0_4dwio6common8FileSink7OptionsE
# 7 _ZNSt17_Function_handlerIFSt10unique_ptrIN8facebook5velox4dwio6common8FileSinkESt14default_deleteIS5_EERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS5_7OptionsEEPSK_E9_M_invokeERKSt9_Any_dataSG_SJ_
# 8 _ZN8facebook5velox4dwio6common8FileSink6createERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS3_7OptionsE
# 9 _ZN6gluten24VeloxParquetDatasourceS38initSinkERKSt13unordered_mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_St4hashIS7_ESt8equal_toIS7_ESaISt4pairIKS7_S7_EEE
# 10 _ZN6gluten22VeloxParquetDatasource4initERKSt13unordered_mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_St4hashIS7_ESt8equal_toIS7_ESaISt4pairIKS7_S7_EEE
# 11 Java_org_apache_gluten_datasource_DatasourceJniWrapper_nativeInitDatasource
# 12 0x00007fae2d018427

24/08/22 09:59:06 ERROR TaskSetManager: Task 0 in stage 2.0 failed 1 times; aborting job
24/08/22 09:59:06 ERROR FileFormatWriter: Aborting job a9b92470-cf6e-4382-b32c-25323810b4bd.
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 2.0 failed 1 times, most recent failure: Lost task 0.0 in stage 2.0 (TID 3) (iZt4n8he1ls2xewpuxsh4yZ executor driver): java.lang.RuntimeException: Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Retriable: False
Expression: pool != nullptr
Function: Impl
File: /home/incubator-gluten/ep/build-velox/build/velox_ep/velox/connectors/hive/storage_adapters/s3fs/S3FileSystem.cpp
Line: 227
Stack trace:
# 0 _ZN8facebook5velox7process10StackTraceC1Ei
# 1 _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2 _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorENS1_22CompileTimeEmptyStringEEEvRKNS1_18VeloxCheckFailArgsET0_
# 3 _ZN8facebook5velox11filesystems11S3WriteFile4ImplC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPN3Aws2S38S3ClientEPNS0_6memory10MemoryPoolE
# 4 _ZN8facebook5velox11filesystems11S3WriteFileC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPN3Aws2S38S3ClientEPNS0_6memory10MemoryPoolE
# 5 _ZN8facebook5velox11filesystems12S3FileSystem16openFileForWriteESt17basic_string_viewIcSt11char_traitsIcEERKNS1_11FileOptionsE
# 6 _ZN8facebook5velox11filesystems24s3WriteFileSinkGeneratorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS0_4dwio6common8FileSink7OptionsE
# 7 _ZNSt17_Function_handlerIFSt10unique_ptrIN8facebook5velox4dwio6common8FileSinkESt14default_deleteIS5_EERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS5_7OptionsEEPSK_E9_M_invokeERKSt9_Any_dataSG_SJ_
# 8 _ZN8facebook5velox4dwio6common8FileSink6createERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS3_7OptionsE
# 9 _ZN6gluten24VeloxParquetDatasourceS38initSinkERKSt13unordered_mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_St4hashIS7_ESt8equal_toIS7_ESaISt4pairIKS7_S7_EEE
# 10 _ZN6gluten22VeloxParquetDatasource4initERKSt13unordered_mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_St4hashIS7_ESt8equal_toIS7_ESaISt4pairIKS7_S7_EEE
# 11 Java_org_apache_gluten_datasource_DatasourceJniWrapper_nativeInitDatasource
# 12 0x00007fae2d018427

Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2672)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2608)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2607)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2607)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1182)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1182)
at scala.Option.foreach(Option.scala:407)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1182)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2860)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2802)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2791)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:952)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2228)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:340)
at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:195)
at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:113)
at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:111)
at org.apache.spark.sql.execution.command.DataWritingCommandExec.executeCollect(commands.scala:125)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:98)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:109)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:169)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:95)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:98)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:94)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:584)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:176)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:584)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:560)
at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:94)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:81)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:79)
at org.apache.spark.sql.Dataset.(Dataset.scala:220)
at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:100)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:97)
at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:622)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:617)
at $line17.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.(:23)
at $line17.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw.(:27)
at $line17.$read$$iw$$iw$$iw$$iw$$iw$$iw.(:29)
at $line17.$read$$iw$$iw$$iw$$iw$$iw.(:31)
at $line17.$read$$iw$$iw$$iw$$iw.(:33)
at $line17.$read$$iw$$iw$$iw.(:35)
at $line17.$read$$iw$$iw.(:37)
at $line17.$read$$iw.(:39)
at $line17.$read.(:41)
at $line17.$read$.(:45)
at $line17.$read$.()
at $line17.$eval$.$print$lzycompute(:7)
at $line17.$eval$.$print(:6)
at $line17.$eval.$print()
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:747)
at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1020)
at scala.tools.nsc.interpreter.IMain.$anonfun$interpret$1(IMain.scala:568)
at scala.reflect.internal.util.ScalaClassLoader.asContext(ScalaClassLoader.scala:36)
at scala.reflect.internal.util.ScalaClassLoader.asContext$(ScalaClassLoader.scala:116)
at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:41)
at scala.tools.nsc.interpreter.IMain.loadAndRunReq$1(IMain.scala:567)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:594)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:564)
at scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:865)
at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:733)
at scala.tools.nsc.interpreter.ILoop.processLine(ILoop.scala:435)
at scala.tools.nsc.interpreter.ILoop.loop(ILoop.scala:456)
at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:239)
at org.apache.spark.repl.Main$.doMain(Main.scala:78)
at org.apache.spark.repl.Main$.main(Main.scala:58)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:958)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.RuntimeException: Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Retriable: False
Expression: pool != nullptr
Function: Impl
File: /home/incubator-gluten/ep/build-velox/build/velox_ep/velox/connectors/hive/storage_adapters/s3fs/S3FileSystem.cpp
Line: 227
Stack trace:
# 0 _ZN8facebook5velox7process10StackTraceC1Ei
# 1 _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2 _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorENS1_22CompileTimeEmptyStringEEEvRKNS1_18VeloxCheckFailArgsET0_
# 3 _ZN8facebook5velox11filesystems11S3WriteFile4ImplC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPN3Aws2S38S3ClientEPNS0_6memory10MemoryPoolE
# 4 _ZN8facebook5velox11filesystems11S3WriteFileC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPN3Aws2S38S3ClientEPNS0_6memory10MemoryPoolE
# 5 _ZN8facebook5velox11filesystems12S3FileSystem16openFileForWriteESt17basic_string_viewIcSt11char_traitsIcEERKNS1_11FileOptionsE
# 6 _ZN8facebook5velox11filesystems24s3WriteFileSinkGeneratorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS0_4dwio6common8FileSink7OptionsE
# 7 _ZNSt17_Function_handlerIFSt10unique_ptrIN8facebook5velox4dwio6common8FileSinkESt14default_deleteIS5_EERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS5_7OptionsEEPSK_E9_M_invokeERKSt9_Any_dataSG_SJ_
# 8 _ZN8facebook5velox4dwio6common8FileSink6createERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS3_7OptionsE
# 9 _ZN6gluten24VeloxParquetDatasourceS38initSinkERKSt13unordered_mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_St4hashIS7_ESt8equal_toIS7_ESaISt4pairIKS7_S7_EEE
# 10 _ZN6gluten22VeloxParquetDatasource4initERKSt13unordered_mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_St4hashIS7_ESt8equal_toIS7_ESaISt4pairIKS7_S7_EEE
# 11 Java_org_apache_gluten_datasource_DatasourceJniWrapper_nativeInitDatasource
# 12 0x00007fae2d018427

at org.apache.gluten.datasource.DatasourceJniWrapper.nativeInitDatasource(Native Method)
at org.apache.gluten.datasource.DatasourceJniWrapper.nativeInitDatasource(DatasourceJniWrapper.java:49)
at org.apache.spark.sql.execution.datasources.velox.VeloxFormatWriterInjects.createOutputWriter(VeloxFormatWriterInjects.scala:70)
at org.apache.spark.sql.execution.datasources.velox.VeloxFormatWriterInjects.createOutputWriter$(VeloxFormatWriterInjects.scala:43)
at org.apache.spark.sql.execution.datasources.velox.VeloxParquetWriterInjects.createOutputWriter(VeloxParquetWriterInjects.scala:26)
at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anon$1.newInstance(ParquetFileFormat.scala:100)
at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.newOutputWriter(FileFormatDataWriter.scala:163)
at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.(FileFormatDataWriter.scala:150)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:417)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$22(FileFormatWriter.scala:351)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:136)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
org.apache.spark.SparkException: Job aborted.
at org.apache.spark.sql.errors.QueryExecutionErrors$.jobAbortedError(QueryExecutionErrors.scala:651)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:376)
at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:195)
at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:113)
at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:111)
at org.apache.spark.sql.execution.command.DataWritingCommandExec.executeCollect(commands.scala:125)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:98)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:109)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:169)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:95)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:98)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:94)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:584)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:176)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:584)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:560)
at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:94)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:81)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:79)
at org.apache.spark.sql.Dataset.(Dataset.scala:220)
at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:100)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:97)
at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:622)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:617)
... 47 elided
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 2.0 failed 1 times, most recent failure: Lost task 0.0 in stage 2.0 (TID 3) (iZt4n8he1ls2xewpuxsh4yZ executor driver): java.lang.RuntimeException: Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Retriable: False
Expression: pool != nullptr
Function: Impl
File: /home/incubator-gluten/ep/build-velox/build/velox_ep/velox/connectors/hive/storage_adapters/s3fs/S3FileSystem.cpp
Line: 227
Stack trace:
# 0 _ZN8facebook5velox7process10StackTraceC1Ei
# 1 _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2 _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorENS1_22CompileTimeEmptyStringEEEvRKNS1_18VeloxCheckFailArgsET0_
# 3 _ZN8facebook5velox11filesystems11S3WriteFile4ImplC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPN3Aws2S38S3ClientEPNS0_6memory10MemoryPoolE
# 4 _ZN8facebook5velox11filesystems11S3WriteFileC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPN3Aws2S38S3ClientEPNS0_6memory10MemoryPoolE
# 5 _ZN8facebook5velox11filesystems12S3FileSystem16openFileForWriteESt17basic_string_viewIcSt11char_traitsIcEERKNS1_11FileOptionsE
# 6 _ZN8facebook5velox11filesystems24s3WriteFileSinkGeneratorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS0_4dwio6common8FileSink7OptionsE
# 7 _ZNSt17_Function_handlerIFSt10unique_ptrIN8facebook5velox4dwio6common8FileSinkESt14default_deleteIS5_EERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS5_7OptionsEEPSK_E9_M_invokeERKSt9_Any_dataSG_SJ_
# 8 _ZN8facebook5velox4dwio6common8FileSink6createERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS3_7OptionsE
# 9 _ZN6gluten24VeloxParquetDatasourceS38initSinkERKSt13unordered_mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_St4hashIS7_ESt8equal_toIS7_ESaISt4pairIKS7_S7_EEE
# 10 _ZN6gluten22VeloxParquetDatasource4initERKSt13unordered_mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_St4hashIS7_ESt8equal_toIS7_ESaISt4pairIKS7_S7_EEE
# 11 Java_org_apache_gluten_datasource_DatasourceJniWrapper_nativeInitDatasource
# 12 0x00007fae2d018427

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

[VL] Pool is null when using s3 cpp sdk to write data into s3 external table

Description

Contributor guide

Assessment

Get new issues in your inbox