apache / apache/arrow

[C++] Unable to use a FIFO as a OSFile path.

Open
#31,186 3 comments 1 reaction 0 assignees View on GitHub
Component: C++ Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

When attempting to use a named pipe or FIFO as an OSFile an error is raised because the file handle is not seekable (lseek() fails).

Having the ability to use a named pipe as an OSFile rather than a PythonFile brings the advantages of no having to interact via the GIL in python.

I propose that there be a way to create an OSFile that represents non-seekable streams, these would be very useful for csv.open_csv().

 

 

 

 

 

**Environment**: Linux
**Reporter**: [Rusty Conover](https://issues.apache.org/jira/browse/ARROW-15738)

**Note**: *This issue was originally created as [ARROW-15738](https://issues.apache.org/jira/browse/ARROW-15738). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by tracing how OSFile validates file handles with lseek() and how that path connects to csv.open_csv(). Determine the design needed for an OSFile backed by a non-seekable FIFO, then verify that named pipes work without requiring Python GIL interaction.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.