apache / apache/arrow

[C++][Python] Filesystem Interface: HadoopFileSystem unable to connect to hdfs - libhdfs not found

Open
#33,913 0 comments 0 reactions 0 assignees View on GitHub
Component: C++ Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

I am running a hadoop distributed filesystem on my local computer and attempting to use the new Filesystem interface with HadoopFileSystem to connect to my hdfs. However, after verifying my environment variables were set correctly and that the libhdfs.so file exists in the specified directory, upon trying to connect, I get the error:

"OSError: Unable to load libhdfs: The specified module could not be found."

Platform: Windows 10
python == 3.11.1
pyarrow==10.0.1
Hadoop/hdfs version == 3.2.2
Namenode port: 'localhost:9000'

![image](https://user-images.githubusercontent.com/67691373/215331205-6dcef419-eca8-452f-87b9-40aadf33f739.png)

I have checked libhdfs.so exists and is not empty at specified location:
![image](https://user-images.githubusercontent.com/67691373/215332258-c9b1046c-2e0a-42e8-b717-c7b8d04abaef.png)

I have also double checked the CLASSPATH variable is set correctly:
![image](https://user-images.githubusercontent.com/67691373/215331453-f62a6585-a3c1-45c3-9dd0-ba8fbac90e4f.png)

Thanks, any help is appreciated.

### Component(s)

C++, Python

Contributor guide

Open the contributing guide

Research direction

Start at the HadoopFileSystem connection path and reproduce the libhdfs loading error on Windows 10 with the reported Python, PyArrow, and Hadoop versions. Check how the configured libhdfs.so path, CLASSPATH, and environment variables are consumed; done means identifying the cause and documenting a verified resolution or supported-configuration limitation.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.