apache / apache/arrow-java

[Java][arrow-jdbc] Ability to customize JdbcConsumer construction

オープン
#224 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る
Type: enhancement
主要言語
Java
スター
94
フォーク
152
平均マージ
3日 16時間
マージ済み PR(30日)
11

説明

### Describe the enhancement requested

I'm working on a project that heavily uses the arrow-jdbc library to convert results from a number of different JDBC drivers to Arrow. When the integration was initially written a bit over 2 years ago it forked `ArrowVectorIterator` to add a number of customization points in the JDBC->Arrow conversion which were missing at the time. I'm updating the project and happily in the intervening time it looks like a lot of this customization is now possible natively via customization points in `JdbcToArrowConfig` - in particular the ability to override `JdbcFieldInfo` per column and most importantly the ability to override a `Function`.

I'd love to fully remove our forked `ArrowVectorIterator` but there's one last customization point we're depending on that I don't see a way to handle currently: The ability to customize the `JdbcConsumer` instances constructed for each column. We depend on overriding these to paper over idiosyncrasies with various JDBC drivers. As an example on the top of my mind this morning: The Snowflake JDBC driver [does not support](https://docs.snowflake.com/en/user-guide/jdbc-api.html#id14) `getBinaryStream` and so is not compatible with the default `BinaryConsumer` and must be overriden with a consumer that uses `getBytes` instead. We also add a layer of wrapping to all of our `JdbcConsumers` to wrap error messages to aid debugging (by adding info on the specific column an error originated from).

Would it be possible to add an extension point to the `JdbcToArrowConfig` to customize this, similar to `jdbcToArrowTypeConverter`? I wish we didn't need it but in practice we've found enough variability across JDBC drivers to necessitate it.

### Component(s)

Java

コントリビューションガイド

コントリビューションガイドを開く

調査の方向性

まず ArrowVectorIterator と JdbcToArrowConfig を読み、次に各列について JdbcConsumer のインスタンスがどのように構築されるかを追跡します。ドライバー固有の consumer と wrapping を可能にする、jdbcToArrowTypeConverter に類似した拡張ポイントを定義し、Snowflake の getBytes workaround をそこから提供できることを検証します。

索引モデルが issue の本文から書いたものです。

評価

技術スタック
java
領域
databases
issue の種類
機能追加
難易度
5/5
見積もり時間
1週間以上
活発さ
停滞
明瞭さ
おおむね明確
初心者へのやさしさ
38/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。