[Python] read an HDFS file by line failed when the open_mode is "rb"
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
Hi there,
I found some problems when I use `{**}fsspec`{**} to read an HDFS file by line when the open_mode is "rb". It works fine when the **open_mode is "r"** or the {**}file is located locally{**}.
some snippets:
```java
import fsspec
hdfs_file_path = "hdfs://xxxxxx"
with fsspec.open(hdfs_file_path, "rb") as f:
# raise UnspportedOperation
f.readline()
```
Error logs:
/opt/conda/lib/python3.7/site-packages/pyarrow/io.pxi in pyarrow.lib.NativeFile.readline()
UnsupportedOperation:
Originally from: https://github.com/fsspec/filesystem_spec/issues/810
**Environment**: System: Ubuntu 18.04
fsspec: 2021.10.1
pyarrow: 6.0.0
**Reporter**: [nero](https://issues.apache.org/jira/browse/ARROW-14787)
**Note**: *This issue was originally created as [ARROW-14787](https://issues.apache.org/jira/browse/ARROW-14787). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start by reproducing the reported fsspec HDFS read with open_mode="rb" and the stated Ubuntu, fsspec 2021.10.1, and pyarrow 6.0.0 environment, then trace the HDFS and pyarrow file-reading path. Done means readline() works for binary HDFS streams without UnsupportedOperation and a regression test covers the behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100