alibaba / alibaba/x-deeplearning
No valid credentials provided
- Dominant language
- PureBasic
- Stars
- 4.3k
- Forks
- 1k
- PR merge metrics
- No merged PRs in 30d
Description
```
root@txyz_18_51:/data/var/nlp/xzy/tdm_mock# python train.py --run_mode=local --config=config.train.json
config: {u'ps': {u'instance_num': 16, u'memory_m': 64000, u'gpu_cores': 0, u'cpu_cores': 16}, u'dependent_dirs': u'/data/var/nlp/xzy/tdm_mock', u'script': u'train.py', u'worker': {u'instance_num': 20, u'memory_m': 100000, u'gpu_cores': 2, u'cpu_cores': 46}, u'max_local_failover_times': 3, u'auto_rebalance': {u'enable': u'false'}, u'min_finish_worker_rate': 100, u'max_failover_times': 3, u'job_name': u'xdl_tdm', u'docker_image': u'registry.cn-hangzhou.aliyuncs.com/xdl/xdl:ubuntu-gpu-mxnet1.3', u'checkpoint': {u'output_dir': u'/user/druid/vrecsys/tdm/tdm_mock_train/checkpoint'}}
mv data/ub_tree.pb data/ub_tree.pb.bak
hadoop fs -get /user/druid/vrecsys/tdm/tree_data/data/userbehavoir_tree.pb data/ub_tree.pb
256
Load successfully, leaf node count:20/01/13 12:07:39 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/01/13 12:07:40 WARN security.UserGroupInformation: PriviledgedActionException as:root (auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
20/01/13 12:07:40 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
20/01/13 12:07:40 WARN security.UserGroupInformation: PriviledgedActionException as:root (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
hdfsGetPathInfo(/user/druid/vrecsys/tdm/tree_data/data/userbehavoir_train_sample.dat_[\d]+): getFileInfo error:
GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "txyz_18_51/127.0.0.1"; destination host is: "adm159133.heracles.sohuno.com":8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
at org.apache.hadoop.ipc.Client.call(Client.java:1508)
at org.apache.hadoop.ipc.Client.call(Client.java:1441)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:260)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
at com.sun.proxy.$Proxy11.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2126)
at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1262)
at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1258)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1258)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1418)
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:718)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:681)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:769)
at org.apache.hadoop.ipc.Client$Connection.access$3000(Client.java:396)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1557)
at org.apache.hadoop.ipc.Client.call(Client.java:1480)
... 17 more
```
在docker环境下运行TDM测试的例子出现以上错误是因为KERBEROS认真的问题嘛,需要怎么样解决呢?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with train.py and config.train.json, then inspect the hadoop fs -get command shown in the report. Reproduce the Docker TDM example and trace the failed connection to the configured HDFS endpoint. Done means identifying the required authentication setup and confirming that the example can retrieve the requested path successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, hadoop, python
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100