[Python] OSError: Unable to load libjvm: /usr/java/latest//lib/amd64/server/libjvm.so
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
```
from pyarrow import fs
hdfs, _ = fs.FileSystem.from_uri('hdfs://haruna')
```
when i use this type of code to create a connect, it raise this error
```
hdfs, _ = fs.FileSystem.from_uri('hdfs://haruna')
File "pyarrow/_fs.pyx", line 463, in pyarrow._fs.FileSystem.from_uri
File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 115, in pyarrow.lib.check_status
OSError: Unable to load libjvm: /usr/java/latest//lib/amd64/server/libjvm.so: cannot open shared object file: No such file or directory
```
environments
```
export JAVA_HOME=/opt/tiger/jdk/jdk8u265-b01
export HADOOP_HOME=/opt/tiger/yarn_deploy/hadoop
export CLASSPATH=`/opt/tiger/yarn_deploy/hadoop/bin/hadoop classpath --glob`
export LD_LIBRARY_PATH=/opt/tiger/yarn_deploy/hadoop/lib/native/
export LIBHDFS_OPTS='-Dhadoop.root.logger=INFO,console'
```
Contributor guide
Research direction
Start with the fs.FileSystem.from_uri entry point in pyarrow/_fs.pyx and trace the reported error through pyarrow/error.pxi. Reproduce the HDFS URI case using the supplied JAVA_HOME, HADOOP_HOME, CLASSPATH, and LD_LIBRARY_PATH settings, then compare the expected JVM library location with the path in the failure. Done means the cause is identified and the issue has a verified fix or documented configuration requirement.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, java, python
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100