fsspec / fsspec/filesystem_spec

'pyarrow._hdfs.HadoopFileSystem' object has no attribute 'host'

Open
#1,870 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
490
Avg merge
2d 3h
Merged PRs (30d)
38

Description

I tried connecting to a HDFS storage, through the default configuation
(core-site.xml). Connecting, plus writing and reading a dataframe worked
find (not shown). However, when attempting to run the code:

"""
import dask.array as da

N = 10_000
rng = da.random.default_rng()
x = rng.random((N, N), chunks=(2000, 2000))
x.to_zarr("hdfs:///user/eriksen/test2.zarr")
"""

It ends up failing with the following issue, which seems to be fsspec related:

File /data/aai/scratch_ssd/eriksen/miniforge3/envs/dask/lib/python3.13/functools.py:1026, in cached_property.get(self, instance, owner)
1024 val = cache.get(self.attrname, _NOT_FOUND)
1025 if val is _NOT_FOUND:
-> 1026 val = self.func(instance)
1027 try:
1028 cache[self.attrname] = val

File /data/aai/scratch_ssd/eriksen/miniforge3/envs/dask/lib/python3.13/site-packages/fsspec/implementations/arrow.py:63, in ArrowFSWrapper.fsid(self)
61 @cached_property
62 def fsid(self):
---> 63 return "hdfs_" + tokenize(self.fs.host, self.fs.port)

AttributeError: 'pyarrow._hdfs.HadoopFileSystem' object has no attribute 'host'

I installed the environment today with Python 3.13 and the following packages:
cloudpickle==3.1.1
dask==2025.5.1
distributed==2025.5.1
fsspec==2025.5.1
pyarrow==20.0.0
toolz==1.0.0
zarr==3.0.8
zict==3.0.0

Please let me know if you need any other information or I should be reporting this
issue elsewhere.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in fsspec/implementations/arrow.py at ArrowFSWrapper.fsid, where the traceback accesses self.fs.host. Reproduce the Dask to_zarr example with the listed Python, fsspec, pyarrow, Dask, and Zarr versions, then inspect the available HadoopFileSystem attributes. Done means the HDFS-backed Zarr write no longer raises this AttributeError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
infrastructure
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.