apache / apache/arrow

[Python][FlightRPC] Is it possible to use pyarrow.dataset as an abstraction over arrow-flight data?

Open
#37,278 6 comments 0 reactions 0 assignees View on GitHub
Component: FlightRPC Component: Python Type: usage
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the usage question you have. Please include as many useful details as possible.

We use an arrow flight based API to export data. It would be very useful to be able to use something like the pyarrow.dataset object over such datasets too.

Right now the dataset wants a directory of files. But it would be useful to be able provide an implementation of an interface that has operations like listing all tables, getting their schemas, reading portions of the tables, etc. So it can be used in front of arrow flight.

If I had such a thing then I could pass a pyarrow.dataset backed by our data to, say, DuckDB and it could efficiently perform queries with row and column filtering.

### Component(s)

FlightRPC, Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.