[C++][Python] Filesystem Interface: HadoopFileSystem unable to connect to hdfs - libhdfs not found
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
I am running a hadoop distributed filesystem on my local computer and attempting to use the new Filesystem interface with HadoopFileSystem to connect to my hdfs. However, after verifying my environment variables were set correctly and that the libhdfs.so file exists in the specified directory, upon trying to connect, I get the error:
"OSError: Unable to load libhdfs: The specified module could not be found."
Platform: Windows 10
python == 3.11.1
pyarrow==10.0.1
Hadoop/hdfs version == 3.2.2
Namenode port: 'localhost:9000'

I have checked libhdfs.so exists and is not empty at specified location:

I have also double checked the CLASSPATH variable is set correctly:

Thanks, any help is appreciated.
### Component(s)
C++, Python
Contributor guide
Research direction
Start at the HadoopFileSystem connection path and reproduce the libhdfs loading error on Windows 10 with the reported Python, PyArrow, and Hadoop versions. Check how the configured libhdfs.so path, CLASSPATH, and environment variables are consumed; done means identifying the cause and documenting a verified resolution or supported-configuration limitation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100