[Python]OSError: Unable to load libjvm. when used in windows
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
Hi all,
I am using pyarrow==7.0.0 to connect hdfs.
It run well with linux,but unfortunately get error in windows.
I have set `JAVA_HOME`, `HADOOPHOME`,`ARROW_LIBHDFS_DIR`
JAVA_HOME=C:\Users\think\Desktop\python-SDK-green\jdk18
HADOOP_HOME=C:\Users\think\Downloads\hadoop-2.10.1.tar\hadoop-2.10.1\hadoop-2.10.1
ARROW_LIBHDFS_DIR=C:\Users\think\Desktop\python-SDK-green\hadoop_client\lib\native;C:\Users\think\Desktop\python-SDK-green\jdk18\jre\bin\server;
when I am using pyarrow.hdfs.connect(), I am getting the error:
```bash
Traceback (most recent call last):
File "C:\Users\think\Desktop\python-SDK-green\python\test.py", line 7, in
data_provider = DataProvider()
File "C:\Users\think\Desktop\python-SDK-green\python\lib\site-packages\nescqdata\MarketData\dataProvider.py", line 15, in __init__
super(DataProvider, self).__init__(dfs)
File "C:\Users\think\Desktop\python-SDK-green\python\lib\site-packages\nescqdata\baseDataProvider.py", line 53, in __init__
self.dfs = pa.hdfs.connect() if dfs is None else dfs
File "C:\Users\think\Desktop\python-SDK-green\python\lib\site-packages\pyarrow\hdfs.py", line 227, in connect
return _connect(
File "C:\Users\think\Desktop\python-SDK-green\python\lib\site-packages\pyarrow\hdfs.py", line 237, in _connect
fs = HadoopFileSystem(host=host, port=port, user=user,
File "C:\Users\think\Desktop\python-SDK-green\python\lib\site-packages\pyarrow\hdfs.py", line 49, in __init__
self._connect(host, port, user, kerb_ticket, extra_conf)
File "pyarrow\_hdfsio.pyx", line 85, in pyarrow._hdfsio.HadoopFileSystem._connect
File "pyarrow\error.pxi", line 114, in pyarrow.lib.check_status
OSError: Unable to load libjvm: �Ҳ���ָ����ģ�顣
```
and by the way, before I got this error,hdfs.py was modified to avoid another problem by add shell=True
```bash
File "C:\Users\think\Desktop\python-SDK-green\python\lib\site-packages\pyarrow\hdfs.py", line 145, in _maybe_set_hadoop_classpath
classpath = _hadoop_classpath_glob(hadoop_bin)
File "C:\Users\think\Desktop\python-SDK-green\python\lib\site-packages\pyarrow\hdfs.py", line 172, in _hadoop_classpath_glob
return subprocess.check_output(hadoop_classpath_args)
File "subprocess.py", line 424, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "subprocess.py", line 505, in run
with Popen(*popenargs, **kwargs) as process:
File "subprocess.py", line 951, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "subprocess.py", line 1420, in _execute_child
hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
OSError: [WinError 193] %1 不是有效的 Win32 应用程序。
```
Thank you !!!
Contributor guide
Research direction
Start with pyarrow/hdfs.py at _maybe_set_hadoop_classpath and HadoopFileSystem._connect, then trace the native libjvm loading reported from pyarrow/_hdfsio.pyx. Reproduce on Windows with the stated JAVA_HOME, HADOOP_HOME, and ARROW_LIBHDFS_DIR values; done means pyarrow.hdfs.connect() loads libjvm without either reported OSError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, java, python
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100