byzer-org / byzer-org/byzer-cli
[bug]HADOOP_HOME and hadoop.home.dir are unset
- Dominant language
- Go
- Stars
- 1
- Forks
- 3
- PR merge metrics
- No merged PRs in 30d
Description
I execute in Windows and find warn exception in the log, but the code can run normally, the log is as follows:
```
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
log4j:WARN No such property [rollingPolicy] in org.apache.log4j.RollingFileAppender.
22/03/30 14:48:26 WARN Shell: Did not find winutils.exe: {}
java.io.FileNotFoundException: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset. -see https://wiki.apache.org/hadoop/WindowsProblems
at org.apache.hadoop.util.Shell.fileNotFoundException(Shell.java:548)
at org.apache.hadoop.util.Shell.getHadoopHomeDir(Shell.java:569)
at org.apache.hadoop.util.Shell.getQualifiedBin(Shell.java:592)
at org.apache.hadoop.util.Shell.(Shell.java:689)
at org.apache.hadoop.util.StringUtils.(StringUtils.java:78)
at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1664)
at org.apache.hadoop.security.SecurityUtil.setConfigurationInternal(SecurityUtil.java:104)
at org.apache.hadoop.security.SecurityUtil.(SecurityUtil.java:88)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:316)
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:304)
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1828)
at org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:710)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:660)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:571)
at org.apache.spark.util.Utils$.$anonfun$getCurrentUserName$1(Utils.scala:2476)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2476)
at org.apache.spark.SparkContext.(SparkContext.scala:314)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2678)
at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:942)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:936)
at streaming.core.strategy.platform.SparkRuntime.createRuntime(SparkRuntime.scala:185)
at streaming.core.strategy.platform.SparkRuntime.(SparkRuntime.scala:63)
at streaming.core.strategy.platform.SparkRuntime$.getOrCreate(SparkRuntime.scala:330)
at streaming.core.strategy.platform.SparkRuntime.getOrCreate(SparkRuntime.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at streaming.core.strategy.platform.PlatformManager$.createRuntimeByPlatform(PlatformManager.scala:237)
at streaming.core.strategy.platform.PlatformManager$.getRuntime(PlatformManager.scala:252)
at streaming.core.strategy.platform.PlatformManager.run(PlatformManager.scala:117)
at streaming.core.StreamingApp$.main(StreamingApp.scala:45)
at streaming.core.StreamingApp.main(StreamingApp.scala)
Caused by: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset.
at org.apache.hadoop.util.Shell.checkHadoopHomeInner(Shell.java:468)
at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:439)
at org.apache.hadoop.util.Shell.(Shell.java:516)
. 31 more
```
Contributor guide
Assessment
This issue has not been assessed yet.