apache / apache/arrow

[Python] `pyarrow.fs.HadoopFileSystem` throws OSError: Unable to load libhdfs

Open
#40,305 2 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

I'm trying to create an HDFS Connection via `pyarrow.fs.HadoopFileSystem`, but unfortunately I get an error:

```{python}
from pyarrow.fs import HadoopFileSystem
hdfs = HadoopFileSystem(
host="localhost",
port=8001,
)
```
> OSError: Unable to load libhdfs: Das angegebene Modul wurde nicht gefunden.

From https://arrow.apache.org/docs/python/filesystems.html#hadoop-distributed-file-system-hdfs I understand that `libhdfs.so` should be located in `%HADOOP_HOME%lib/native/`, which is the case. I also set the `CLASSPATH` environment variable to `%HADOOP_HOME%/bin/hadoop`.

What am I missing?

I use pyarrow==15.0.0.

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.