linkedin / linkedin/coral

bootRun: Could not locate executable null\bin\winutils.exe in the Hadoop binaries And throwing NullPointerException when initLocalMetastore()

Open
#545 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
906
Forks
219
Avg merge
2d 13h
Merged PRs (30d)
2

Description

I encountered some errors getting started with Coral. I'm using jdk8 and Python3.8.
Under coral-service, I executed: ../gardlew bootRun --args='--spring.profiles.active=localMetastore'
then follows the logs:

Task :coral-service:bootRun

. ____ _ __ _ _
/\ / ' __ _ () __ __ _ \ \ \
( ( )_
_ | '_ | '| | ' / ` | \ \ \
\/ )| |)| | | | | || (| | ) ) ) )
' |
| .__|| ||| |_, | / / / /
=========|
|==============|/=////
:: Spring Boot :: (v2.4.0)

2025-01-09 18:34:04.871 INFO 3216 --- [ main] c.l.c.c.CoralServiceApplication : Starting CoralServiceApplication using Java 1.8.0_291 on LAPTOP-2S6ONLIK with PID 3216 (E:\project\coral\coral-service\build\classes\java\main started by xh254 in E:\project\coral\coral-service)
2025-01-09 18:34:04.873 INFO 3216 --- [ main] c.l.c.c.CoralServiceApplication : The following profiles are active: localMetastore
2025-01-09 18:34:06.194 INFO 3216 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http)
2025-01-09 18:34:06.204 INFO 3216 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
2025-01-09 18:34:06.204 INFO 3216 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.39]
2025-01-09 18:34:06.337 INFO 3216 --- [ main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2025-01-09 18:34:06.337 INFO 3216 --- [ main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 1330 ms
2025-01-09 18:34:06.474 INFO 3216 --- [ main] o.s.s.concurrent.ThreadPoolTaskExecutor : Initializing ExecutorService 'applicationTaskExecutor'
2025-01-09 18:34:06.557 WARN 3216 --- [ main] ion$DefaultTemplateResolverConfiguration : Cannot find template location: classpath:/templates/ (please add some templates or check your Thymeleaf configuration)
2025-01-09 18:34:06.614 INFO 3216 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path ''
2025-01-09 18:34:06.631 ERROR 3216 --- [ main] org.apache.hadoop.util.Shell : Failed to locate the winutils binary in the hadoop binary path

java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:356) [hadoop-common-2.7.0.jar:na]
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:371) [hadoop-common-2.7.0.jar:na]
at org.apache.hadoop.util.Shell.(Shell.java:364) [hadoop-common-2.7.0.jar:na]
at org.apache.hadoop.hive.conf.HiveConf$ConfVars.findHadoopBinary(HiveConf.java:2333) [hive-common-1.2.2.jar:1.2.2]
at org.apache.hadoop.hive.conf.HiveConf$ConfVars.(HiveConf.java:368) [hive-common-1.2.2.jar:1.2.2]
at org.apache.hadoop.hive.conf.HiveConf.(HiveConf.java:108) [hive-common-1.2.2.jar:1.2.2]
at com.linkedin.coral.coralservice.utils.CoralProvider.loadResourceHiveConf(CoralProvider.java:86) [main/:na]
at com.linkedin.coral.coralservice.utils.CoralProvider.initLocalMetastore(CoralProvider.java:53) [main/:na]
at com.linkedin.coral.coralservice.controller.TranslationControllerLocal.onApplicationEvent(TranslationControllerLocal.java:36) [main/:na]
at com.linkedin.coral.coralservice.controller.TranslationControllerLocal.onApplicationEvent(TranslationControllerLocal.java:27) [main/:na]
at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:203) [spring-context-5.3.1.jar:5.3.1]
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:196) [spring-context-5.3.1.jar:5.3.1]
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:161) [spring-context-5.3.1.jar:5.3.1]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:426) [spring-context-5.3.1.jar:5.3.1]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:383) [spring-context-5.3.1.jar:5.3.1]
at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:945) [spring-context-5.3.1.jar:5.3.1]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:591) [spring-context-5.3.1.jar:5.3.1]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:144) [spring-boot-2.4.0.jar:2.4.0]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:767) [spring-boot-2.4.0.jar:2.4.0]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:759) [spring-boot-2.4.0.jar:2.4.0]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:426) [spring-boot-2.4.0.jar:2.4.0]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:326) [spring-boot-2.4.0.jar:2.4.0]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1309) [spring-boot-2.4.0.jar:2.4.0]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1298) [spring-boot-2.4.0.jar:2.4.0]
at com.linkedin.coral.coralservice.CoralServiceApplication.main(CoralServiceApplication.java:19) [main/:na]

2025-01-09 18:34:06.798 INFO 3216 --- [ main] c.l.c.c.CoralServiceApplication : Temp Workspace: C:\Users\xh254\AppData\Local\Temp/coral/service/d1ac1ebb-91cc-40b4-8614-928923d6c76a
2025-01-09 18:34:06.871 WARN 3216 --- [ main] org.apache.hadoop.util.NativeCodeLoader : Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2025-01-09 18:34:06.897 INFO 3216 --- [ main] o.a.hadoop.hive.metastore.HiveMetaStore : 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
2025-01-09 18:34:06.941 INFO 3216 --- [ main] o.a.hadoop.hive.metastore.ObjectStore : ObjectStore, initialize called
log4j:WARN No appenders could be found for logger (DataNucleus.General).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
2025-01-09 18:34:07.814 INFO 3216 --- [ main] o.a.hadoop.hive.metastore.ObjectStore : Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
2025-01-09 18:34:08.896 INFO 3216 --- [ main] o.a.h.hive.metastore.MetaStoreDirectSql : Using direct SQL, underlying DB is DERBY
2025-01-09 18:34:08.897 INFO 3216 --- [ main] o.a.hadoop.hive.metastore.ObjectStore : Initialized ObjectStore
2025-01-09 18:34:08.953 WARN 3216 --- [ main] o.a.hadoop.hive.metastore.ObjectStore : Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
2025-01-09 18:34:09.044 WARN 3216 --- [ main] o.a.hadoop.hive.metastore.ObjectStore : Failed to get database default, returning NoSuchObjectException
2025-01-09 18:34:09.138 INFO 3216 --- [ main] o.a.hadoop.hive.metastore.HiveMetaStore : Added admin role in metastore
2025-01-09 18:34:09.141 INFO 3216 --- [ main] o.a.hadoop.hive.metastore.HiveMetaStore : Added public role in metastore
2025-01-09 18:34:09.189 INFO 3216 --- [ main] o.a.hadoop.hive.metastore.HiveMetaStore : No user is added in admin role, since config is empty
2025-01-09 18:34:09.247 INFO 3216 --- [ main] o.a.hadoop.hive.metastore.HiveMetaStore : 0: get_all_databases
2025-01-09 18:34:09.247 INFO 3216 --- [ main] o.a.h.h.metastore.HiveMetaStore.audit : ugi=xh254 ip=unknown-ip-addr cmd=get_all_databases
2025-01-09 18:34:09.258 INFO 3216 --- [ main] o.a.hadoop.hive.metastore.HiveMetaStore : 0: get_functions: db=default pat=*
2025-01-09 18:34:09.259 INFO 3216 --- [ main] o.a.h.h.metastore.HiveMetaStore.audit : ugi=xh254 ip=unknown-ip-addr cmd=get_functions: db=default pat=*
java.lang.RuntimeException: java.lang.NullPointerException
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:466)
at com.linkedin.coral.coralservice.utils.CoralProvider.initLocalMetastore(CoralProvider.java:64)
at com.linkedin.coral.coralservice.controller.TranslationControllerLocal.onApplicationEvent(TranslationControllerLocal.java:36)
at com.linkedin.coral.coralservice.controller.TranslationControllerLocal.onApplicationEvent(TranslationControllerLocal.java:27)
at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:203)
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:196)
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:161)
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:426)
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:383)
at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:945)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:591)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:144)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:767)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:759)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:426)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:326)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1309)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1298)
at com.linkedin.coral.coralservice.CoralServiceApplication.main(CoralServiceApplication.java:19)
Caused by: java.lang.NullPointerException
Caused by: java.lang.NullPointerException

at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:483)
at org.apache.hadoop.util.Shell.run(Shell.java:456)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:815)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:798)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:728)
at org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:486)
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:527)
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:504)
at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:305)
at org.apache.hadoop.hive.ql.exec.Utilities.createDirsWithPermission(Utilities.java:3678)
at org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:597)
at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:554)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:508)
... 19 more

2025-01-09 18:34:09.341 INFO 3216 --- [ main] c.l.c.c.CoralServiceApplication : Started CoralServiceApplication in 4.765 seconds (JVM running for 5.318)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the localMetastore startup from coral-service with the reported Java 8 command, then inspect CoralProvider.java at initLocalMetastore() and loadResourceHiveConf(), alongside TranslationControllerLocal.java. Trace the Hadoop winutils path and the SessionState startup failure on Windows; done means bootRun initializes the local metastore without the missing executable or NullPointerException.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, java, spring-boot
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.