influxdata / influxdata/influxdb
Support apache arrow based data source for Flux
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
__Proposal:__
Enhance Flux to support Apache Arrow (https://arrow.apache.org/) format as an additional data source, so that other data sources producing Apache Arrow could be connected to Flux for computation.
__Current behavior:__
Currently there is a `sql` package allowing users to connect to several relational databases, and custom data source seems to be supported according to the video here (https://www.youtube.com/watch?v=lqUPBEv4X4M), but I don't find enough documentation on how this could be done yet.
__Desired behavior:__
1) Since custom data source for Flux seems to be supported, I would like to see more developer documentation describing how this could be done, for example:
1.1) what kind of data source could be connected based on the current data model
1.2) what the extension APIs are for custom data source, how they could be used to create a new data source
2) I am not familiar with Flux/the extension API yet but since Influxdb uses Apache Arrow under the hood, I think there could be possibility that Apache Arrow based data source could be connected and consumed by Flux if the data is presented in some way Flux accepts. There are two major ways for consumption as far as I can see:
2.1) data source stores Arrow IPC file formats (or Parquet) on disk, and Flux can load the data file on disk and run computation on top of it. This is similar with loading a csv file, but with less resource for deserialization if Apache Arrow is used.
2.2) data source that supports Apache Arrow Flight protocol (https://arrow.apache.org/blog/2019/10/13/introducing-arrow-flight/). This will enable remote data sources just like the `sql` package Flux currently supports.
__Alternatives considered:__
It will be great if the extension API is better documented so that developers could try different ways to connect Flux to different data sources if it sees fit.
__Use case:__
Why is this important (helps with prioritizing requests)?
Since more data systems starts to support Apache Arrow, it will enable Flux/Influxdb users to connect to lots of data sources with minimal effort.
Contributor guide
Assessment
This issue has not been assessed yet.