apache / apache/gluten

[VL] Hadoop Kerberos support

Open
#2,744 13 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description
使用的jar:**gluten-velox-bundle-spark3.2_2.12-centos_7-1.0.0.jar**

我使用的hdfs集群是开启了kerberos认证的,按照文档所说做了配置
--conf spark.executorEnv.LIBHDFS3_CONF="hdfs-client.xml"
--files /path/hdfs-client.xml,/tmp/krb5cc_xxx
--conf spark.executorEnv.KRB5CCNAME=krb5cc_xxx

hdfs-client.xml
``
` `hadoop.security.authentication``
``kerberos``
``

使用yarn client 模式提交一个简单的任务运行失败,希望能给我一些建议和帮助

```
Reason: Unable to connect to HDFS: ossbHA, got error: HdfsRpcException: Failed to invoke RPC call "getFsStats" on server "xxxxx:8020" Caused by: HdfsIOException: Cannot initialize client (2): Unknown SASL mechanism.
Retriable: False
Expression: hdfsClient_ != nullptr
Context: Split [file hdfs://ossbHA/data/sparkTest/test.snappy.parquet 0 - 113270130] Task Gluten stage-1 task-1
Top-Level Context: Same as context.
Function: Impl
File: /WorkSpace/gluten/ep/build-velox/build/velox_ep/velox/connectors/hive/storage_adapters/hdfs/HdfsFileSystem.cpp
Line: 48
Stack trace:
# 0 _ZN8facebook5velox7process10StackTraceC1Ei
# 1 _ZN8facebook5velox14VeloxExceptionC1EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bNS1_4TypeES7_
# 2 _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKSsEEvRKNS1_18VeloxCheckFailArgsET0_
# 3 _ZN8facebook5velox11filesystems14HdfsFileSystemC2ERKSt10shared_ptrIKNS0_6ConfigEERKNS1_19HdfsServiceEndpointE
# 4 _ZN5folly15basic_once_flagINS_15SharedMutexImplILb0EvSt6atomicNS_24SharedMutexPolicyDefaultEEES2_E14call_once_slowIZNK8facebook5velox11filesystemsUlSt10shared_ptrIKNS8_6ConfigEESt17basic_string_viewIcSt11char_traitsIcEEE_clESD_SH_EUlvE_JEEEvOT_DpOT0_
# 5 _ZNK8facebook5velox11filesystemsUlSt10shared_ptrIKNS0_6ConfigEESt17basic_string_viewIcSt11char_traitsIcEEE_clES5_S9_.isra.0
# 6 _ZNSt17_Function_handlerIFSt10shared_ptrIN8facebook5velox11filesystems10FileSystemEES0_IKNS2_6ConfigEESt17basic_string_viewIcSt11char_traitsIcEEENS3_UlS8_SC_E_EE9_M_invokeERKSt9_Any_dataOS8_OSC_
# 7 _ZN8facebook5velox11filesystems13getFileSystemESt17basic_string_viewIcSt11char_traitsIcEESt10shared_ptrIKNS0_6ConfigEE
# 8 _ZN8facebook5velox19FileHandleGeneratorclERKSs
# 9 _ZN8facebook5velox13CachedFactoryISsNS0_10FileHandleENS0_19FileHandleGeneratorENS0_15FileHandleSizerESt8equal_toISsESt4hashISsEE8generateERKSs
# 10 _ZN8facebook5velox9connector4hive14HiveDataSource8addSplitESt10shared_ptrINS1_14ConnectorSplitEE
# 11 _ZN8facebook5velox4exec9TableScan9getOutputEv
# 12 _ZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEE
# 13 _ZN8facebook5velox4exec6Driver4nextERSt10shared_ptrINS1_13BlockingStateEE
# 14 _ZN8facebook5velox4exec4Task4nextEPN5folly10SemiFutureINS3_4UnitEEE
# 15 _ZN6gluten24WholeStageResultIterator4nextEv
# 16 Java_io_glutenproject_vectorized_ColumnarBatchOutIterator_nativeHasNext
# 17 0x00007fb0e85a5a34

at io.glutenproject.vectorized.ColumnarBatchOutIterator.nativeHasNext(Native Method)
at io.glutenproject.vectorized.ColumnarBatchOutIterator.hasNextInternal(ColumnarBatchOutIterator.java:47)
at io.glutenproject.vectorized.GeneralOutIterator.hasNext(GeneralOutIterator.java:37)
at io.glutenproject.backendsapi.velox.IteratorHandler$$anon$2.hasNext(IteratorHandler.scala:240)
at io.glutenproject.vectorized.CloseableColumnBatchIterator.hasNext(CloseableColumnBatchIterator.scala:41)
at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
at org.apache.spark.shuffle.ColumnarShuffleWriter.internalWrite(ColumnarShuffleWriter.scala:104)
at org.apache.spark.shuffle.ColumnarShuffleWriter.write(ColumnarShuffleWriter.scala:204)
at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
at org.apache.spark.scheduler.Task.run(Task.scala:131)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1462)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

```
### Spark version

spark3.2.1-hadoop2.7

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with Velox/connectors/hive/storage_adapters/hdfs/HdfsFileSystem.cpp at line 48 and review how the HDFS client is initialized for the reported Spark 3.2.1/Hadoop 2.7 yarn-client configuration. Reproduce the task using hdfs-client.xml, the Kerberos credential cache, and the listed executor environment settings. Done means the HDFS read completes without the Unknown SASL mechanism error.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.