influxdata / influxdata/influxdb

Support apache arrow based data source for Flux

Open
#18,315 1 comment 0 reactions 2 assignees Claimed by @nathanielc View on GitHub
area/flux
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

__Proposal:__
Enhance Flux to support Apache Arrow (https://arrow.apache.org/) format as an additional data source, so that other data sources producing Apache Arrow could be connected to Flux for computation.

__Current behavior:__
Currently there is a `sql` package allowing users to connect to several relational databases, and custom data source seems to be supported according to the video here (https://www.youtube.com/watch?v=lqUPBEv4X4M), but I don't find enough documentation on how this could be done yet.

__Desired behavior:__
1) Since custom data source for Flux seems to be supported, I would like to see more developer documentation describing how this could be done, for example:
1.1) what kind of data source could be connected based on the current data model
1.2) what the extension APIs are for custom data source, how they could be used to create a new data source
2) I am not familiar with Flux/the extension API yet but since Influxdb uses Apache Arrow under the hood, I think there could be possibility that Apache Arrow based data source could be connected and consumed by Flux if the data is presented in some way Flux accepts. There are two major ways for consumption as far as I can see:
2.1) data source stores Arrow IPC file formats (or Parquet) on disk, and Flux can load the data file on disk and run computation on top of it. This is similar with loading a csv file, but with less resource for deserialization if Apache Arrow is used.
2.2) data source that supports Apache Arrow Flight protocol (https://arrow.apache.org/blog/2019/10/13/introducing-arrow-flight/). This will enable remote data sources just like the `sql` package Flux currently supports.

__Alternatives considered:__
It will be great if the extension API is better documented so that developers could try different ways to connect Flux to different data sources if it sees fit.

__Use case:__
Why is this important (helps with prioritizing requests)?
Since more data systems starts to support Apache Arrow, it will enable Flux/Influxdb users to connect to lots of data sources with minimal effort.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.