apache / apache/arrow

[Python] pyarrow.fs.HadoopFileSystem cannot access Azure Data Lake (ADLS)

Open
#26,807 5 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

It's not possible to open a ``abfs://`` or `abfss://` URI with the pyarrow.fs.HadoopFileSystem.

Using HadoopFileSystem.from_uri(path) does not work and libhdfs will throw an error saying that the authority is invalid (I checked that this is because the string is empty).

Note that the legacy pyarrow.hdfs.HadoopFileSystem interface works by doing for example:
- pyarrow.hdfs.HadoopFileSystem(host="abfs://xxx@xxx.dfs.core.windows.net")
- pyarrow.hdfs.connect(host="abfs://xxx@xxx.dfs.core.windows.net")

and I believe the new interface should work too by passing the full URI as "host" to `pyarrow.fs.HadoopFileSystem` constructor. However, the constructor wrongly prepends "hdfs://" at the beginning: 

**Reporter**: [Juan Galvez](https://issues.apache.org/jira/browse/ARROW-10872)

**Note**: *This issue was originally created as [ARROW-10872](https://issues.apache.org/jira/browse/ARROW-10872). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start in python/pyarrow/_hdfs.pyx at the constructor linked in the issue, then trace HadoopFileSystem.from_uri(path) for abfs:// and abfss:// URIs. Compare this with the legacy pyarrow.hdfs interface examples in the report. Done means the new pyarrow.fs.HadoopFileSystem interface accepts the full Azure Data Lake URI without producing an invalid authority.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, hadoop, python
Domain
cloud, data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.