apache / apache/arrow-java

[Java][arrow-jdbc] Ability to customize JdbcConsumer construction

Abierto
#224 1 comentario 0 reacciones 0 asignados Ver en GitHub
Type: enhancement
Lenguaje dominante
Java
Estrellas
94
Forks
152
Merge medio
3 d 16 h
PR fusionados (30 d)
11

Descripción

### Describe the enhancement requested

I'm working on a project that heavily uses the arrow-jdbc library to convert results from a number of different JDBC drivers to Arrow. When the integration was initially written a bit over 2 years ago it forked `ArrowVectorIterator` to add a number of customization points in the JDBC->Arrow conversion which were missing at the time. I'm updating the project and happily in the intervening time it looks like a lot of this customization is now possible natively via customization points in `JdbcToArrowConfig` - in particular the ability to override `JdbcFieldInfo` per column and most importantly the ability to override a `Function`.

I'd love to fully remove our forked `ArrowVectorIterator` but there's one last customization point we're depending on that I don't see a way to handle currently: The ability to customize the `JdbcConsumer` instances constructed for each column. We depend on overriding these to paper over idiosyncrasies with various JDBC drivers. As an example on the top of my mind this morning: The Snowflake JDBC driver [does not support](https://docs.snowflake.com/en/user-guide/jdbc-api.html#id14) `getBinaryStream` and so is not compatible with the default `BinaryConsumer` and must be overriden with a consumer that uses `getBytes` instead. We also add a layer of wrapping to all of our `JdbcConsumers` to wrap error messages to aid debugging (by adding info on the specific column an error originated from).

Would it be possible to add an extension point to the `JdbcToArrowConfig` to customize this, similar to `jdbcToArrowTypeConverter`? I wish we didn't need it but in practice we've found enough variability across JDBC drivers to necessitate it.

### Component(s)

Java

Guía de contribución

Abrir la guía de contribución

Línea de trabajo

Empieza leyendo ArrowVectorIterator y JdbcToArrowConfig; después, sigue cómo se construyen las instancias de JdbcConsumer para cada columna. Define un punto de extensión análogo a jdbcToArrowTypeConverter que permita consumers específicos del driver y wrapping, y verifica que el workaround de getBytes de Snowflake pueda proporcionarse a través de él.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
java
Área
databases
Tipo de issue
Nueva funcionalidad
Dificultad
5/5
Tiempo estimado
Más de una semana
Estado de actividad
Estancado
Claridad
Bastante claro
Aptitud para principiantes
38/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.