apache / apache/arrow-java

[Java][arrow-jdbc] Ability to customize JdbcConsumer construction

未關閉
#224 1 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
Type: enhancement
主要語言
Java
星號
94
分支
152
平均合併
3 天 16 小時
30 天內合併 PR
11

描述

### Describe the enhancement requested

I'm working on a project that heavily uses the arrow-jdbc library to convert results from a number of different JDBC drivers to Arrow. When the integration was initially written a bit over 2 years ago it forked `ArrowVectorIterator` to add a number of customization points in the JDBC->Arrow conversion which were missing at the time. I'm updating the project and happily in the intervening time it looks like a lot of this customization is now possible natively via customization points in `JdbcToArrowConfig` - in particular the ability to override `JdbcFieldInfo` per column and most importantly the ability to override a `Function`.

I'd love to fully remove our forked `ArrowVectorIterator` but there's one last customization point we're depending on that I don't see a way to handle currently: The ability to customize the `JdbcConsumer` instances constructed for each column. We depend on overriding these to paper over idiosyncrasies with various JDBC drivers. As an example on the top of my mind this morning: The Snowflake JDBC driver [does not support](https://docs.snowflake.com/en/user-guide/jdbc-api.html#id14) `getBinaryStream` and so is not compatible with the default `BinaryConsumer` and must be overriden with a consumer that uses `getBytes` instead. We also add a layer of wrapping to all of our `JdbcConsumers` to wrap error messages to aid debugging (by adding info on the specific column an error originated from).

Would it be possible to add an extension point to the `JdbcToArrowConfig` to customize this, similar to `jdbcToArrowTypeConverter`? I wish we didn't need it but in practice we've found enough variability across JDBC drivers to necessitate it.

### Component(s)

Java

貢獻指南

開啟貢獻指南

研究方向

先閱讀 ArrowVectorIterator 和 JdbcToArrowConfig,然後追蹤每個欄位的 JdbcConsumer 執行個體是如何建構的。定義一個類似於 jdbcToArrowTypeConverter 的擴充點,使其支援特定於驅動程式的 consumer 和 wrapping,並驗證 Snowflake 的 getBytes workaround 可以透過該擴充點提供。

由索引模型根據 Issue 內容生成。

評估

技術堆疊
java
領域
databases
Issue 類型
功能
難度
5/5
預估耗時
一週以上
活躍度
停滯
描述清晰度
基本清楚
新手友好度
38/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。