flight-sql-jdbc-driver: add option to rely on Arrow Flight SQL Bulk Ingestion for batched inserts
- 主要語言
- Java
- 星號
- 94
- 分支
- 152
- 平均合併
- 3 天 16 小時
- 30 天內合併 PR
- 11
描述
### Describe the enhancement requested
Arrow Flight SQL has a feature for ingesting massive datasets, Bulk Ingestion: https://github.com/apache/arrow/issues/38255
It would be beneficial to use those special RPC methods for batched prepared statement calls when the prepared statement is strictly for inserting data.
E.g. when Spark is used for writing data, it generates a simple SQL query like `"INSERT INTO table(field1, field2, ...) VALUES (?, ?, ...)"`, creates a prepared statement, and then uses the prepared statement update RPC method to insert the rows of the dataset. If this feature is implemented, it would be possible for the driver to instead use the `DoPut(CommandStatementIngest)` command associated with the Bulk Ingestion feature instead.
There are Arrow Flight SQL server implementations that work like this: when a `DoAction(ActionCreatePreparedStatementRequest)` is executed, the server creates up to two versions of the data structure underlying the instance of the PreparedStatement. One is a handle to a full-scale query engine execution procedure (e.g. DataFusion's logical plan), and another is a handle to a very simple procedure that just stores the received record batches in the storage - of course, the second procedure is only possible to be generated when the query is of a certain form; like the one used by Spark. The point is that this simple procedure also works without overheads associated with the more general interface of prepared statement API - for example, it does not need to do a transposition of PreparedStatement parameters into record batches.
I think that it should be possible to move this logic for deciding to use Bulk Ingestion into the jdbc driver.
Usecase for this integration is this: developers of Arrow Flight SQL servers could implement bulk ingestion command handlers and avoid implementing special logic for handling batched inserts in a special manner. Then the client would use this newly introduced driver option to allow the driver to decide to use the bulk ingestion RPC methods for inserting data.
貢獻指南
研究方向
首先檢視 JDBC 驅動程式的 prepared statement 更新路徑與 Flight SQL Bulk Ingestion RPC,尤其是 DoPut(CommandStatementIngest) 和 ActionCreatePreparedStatementRequest。定義選項如何識別符合條件的僅插入批次陳述式,並透過 bulk ingestion 對其進行路由,同時相容於其他 prepared statement。
由索引模型根據 Issue 內容生成。
評估
- 技術堆疊
- java
- 領域
- database
- Issue 類型
- 功能
- 難度
- 5/5
- 預估耗時
- 一週以上
- 活躍度
- 停滯
- 描述清晰度
- 需要釐清
- 新手友好度
- 35/100