[Feature Request]: When Hop adds SQL and thin client driver capability back, like a prior product, use Arrow Flight SQL
- Dominant language
- Java
- Stars
- 1.5k
- Forks
- 476
- Avg merge
- 18h 32m
- Merged PRs (30d)
- 216
Description
### What would you like to happen?
Performing SQL upon pipelines was magical. When looking to do this with Calcite, please look at using Apache Arrow Flight SQL for endpoints to enable faster data transfer between JVM and clients.
Dremio is doing this now and offers a new JDBC Arrow Flight driver in addition to the older JDBC driver.
Their purpose of doing that is to circumvent the problems JDBC had with shipping large amounts of data.
https://docs.dremio.com/software/drivers/arrow-flight-sql-jdbc-driver/
"The JDBC driver for Arrow Flight SQL is an open-source driver that is based on the specifications for the Java Database Connectivity (JDBC) API. However, the Flight JDBC driver uses Apache Arrow, so it is able to move large amounts of data faster, in part because it does not need to serialize and then deserialize data.
This driver solves a problem that is common to many BI tools that access databases through JDBC. These tools bundle a different JDBC driver for each type of database they support, because each of these databases has their own proprietary driver. Bundling multiple JDBC drivers for multiple databases can be difficult to maintain, and responding to support issues for multiple drivers can be costly. Now, provided that a database has an Apache Arrow Flight SQL endpoint enabled, the JDBC driver for Arrow Flight SQL can connect to it." - Dremio documentation for JDBC Driver for Arrow Flight SQL.
Official Documentation for Arrow Flight SQL RPC spec here
https://arrow.apache.org/docs/format/FlightSql.html
I read everything that there is out there on this kind of data movement. The spec is currently experimental, but from my findings, it is gaining rapid adoption. This will be the industry standard in about a year. The whole world is using Arrow in critical softwares to accelerate data accessibility without copy and conversions to whatever the local VM of a language is This is the right bet for the future.
### Issue Priority
Priority: 3
### Issue Component
Component: Metadata
Contributor guide
Research direction
The issue names Calcite and Apache Arrow Flight SQL but no repository files, tests, or entry points. Start by mapping any existing SQL and thin-client capability in the Metadata component, then compare its endpoint and client needs with the official Flight SQL specification. Done would require an agreed scope and acceptance tests, which this issue does not define.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- api, backend-api-design, data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100