apache / apache/arrow

[Python] OSError: Unable to load libjvm: /usr/java/latest//lib/amd64/server/libjvm.so

Open
#14,686 10 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

```
from pyarrow import fs

hdfs, _ = fs.FileSystem.from_uri('hdfs://haruna')
```
when i use this type of code to create a connect, it raise this error
```
hdfs, _ = fs.FileSystem.from_uri('hdfs://haruna')
File "pyarrow/_fs.pyx", line 463, in pyarrow._fs.FileSystem.from_uri
File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 115, in pyarrow.lib.check_status
OSError: Unable to load libjvm: /usr/java/latest//lib/amd64/server/libjvm.so: cannot open shared object file: No such file or directory

```

environments
```
export JAVA_HOME=/opt/tiger/jdk/jdk8u265-b01
export HADOOP_HOME=/opt/tiger/yarn_deploy/hadoop
export CLASSPATH=`/opt/tiger/yarn_deploy/hadoop/bin/hadoop classpath --glob`
export LD_LIBRARY_PATH=/opt/tiger/yarn_deploy/hadoop/lib/native/
export LIBHDFS_OPTS='-Dhadoop.root.logger=INFO,console'
```

Contributor guide

Open the contributing guide

Research direction

Start with the fs.FileSystem.from_uri entry point in pyarrow/_fs.pyx and trace the reported error through pyarrow/error.pxi. Reproduce the HDFS URI case using the supplied JAVA_HOME, HADOOP_HOME, CLASSPATH, and LD_LIBRARY_PATH settings, then compare the expected JVM library location with the path in the failure. Done means the cause is identified and the issue has a verified fix or documented configuration requirement.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, java, python
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.