apache / apache/arrow

[Python] read an HDFS file by line failed when the open_mode is "rb"

Open
#30,322 2 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

Hi there,

I found some problems when I use `{**}fsspec`{**} to read an HDFS file by line when the open_mode is "rb". It works fine when the **open_mode is "r"** or the {**}file is located locally{**}.

some snippets:
```java

import fsspec
hdfs_file_path = "hdfs://xxxxxx"
with fsspec.open(hdfs_file_path, "rb") as f:
# raise UnspportedOperation
f.readline()
```
 

Error logs:
 
/opt/conda/lib/python3.7/site-packages/pyarrow/io.pxi in pyarrow.lib.NativeFile.readline()

UnsupportedOperation:

Originally from: https://github.com/fsspec/filesystem_spec/issues/810

**Environment**: System: Ubuntu 18.04
fsspec: 2021.10.1
pyarrow: 6.0.0
**Reporter**: [nero](https://issues.apache.org/jira/browse/ARROW-14787)

**Note**: *This issue was originally created as [ARROW-14787](https://issues.apache.org/jira/browse/ARROW-14787). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the reported fsspec HDFS read with open_mode="rb" and the stated Ubuntu, fsspec 2021.10.1, and pyarrow 6.0.0 environment, then trace the HDFS and pyarrow file-reading path. Done means readline() works for binary HDFS streams without UnsupportedOperation and a regression test covers the behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.