[Java] Implement converter between Arrow record batches and Avro records
- Langage dominant
- Java
- Étoiles
- 94
- Forks
- 152
- Merge moyen
- 3 j 16 h
- PR mergées (30 j)
- 11
Description
It would be useful for applications which need convert Avro data to Arrow data.
This is an adapter which convert data with existing API (like JDBC adapter) rather than a native reader (like orc).
We implement this function through Avro java project, receiving param like Decoder/Schema/DatumReader of Avro and return VectorSchemaRoot. For each data type we have a consumer class as below to get Avro data and write it into vector to avoid boxing/unboxing (e.g. GenericRecord#get returns Object)
```java
public class AvroIntConsumer implements Consumer {
private final IntWriter writer;
public AvroIntConsumer(IntVector vector)
{ this.writer = new IntWriterImpl(vector); }
@Override
public void consume(Decoder decoder) throws IOException
{ writer.writeInt(decoder.readInt()); writer.setPosition(writer.getPosition() + 1); }
```
We intended to support primitive and complex types (null value represented via unions type with null type), size limit and field selection could be optional for users.
**Reporter**: [Ji Liu](https://issues.apache.org/jira/browse/ARROW-5845) / @tianchen92
#### Subtasks:
- [X] [[Java] Create Avro adapter module and add dependencies](https://github.com/apache/arrow/issues/22265)
- [X] [[Java] Initial implement to convert Avro record with primitive types](https://github.com/apache/arrow/issues/22278)
- [X] [[Java] Avro adapter implement simple Record type ](https://github.com/apache/arrow/issues/22396)
- [X] [[Java] Avro adapter support convert nullable value](https://github.com/apache/arrow/issues/22439)
- [X] [[Java] Avro adapter implement unions type](https://github.com/apache/arrow/issues/22496)
- [X] [[Java] Avro adapter avoid potential resource leak.](https://github.com/apache/arrow/issues/22590)
- [X] [[Java] Add API to avro adapter to limit number of rows returned at a time.](https://github.com/apache/arrow/issues/22608)
- [X] [[Java] Avro adapter implement Array/Map/Fixed type](https://github.com/apache/arrow/issues/22649)
- [X] [[Java] Avro adapter implement Enum type and nested Record type](https://github.com/apache/arrow/issues/16862)
- [X] [[Java] Add benchmark and large fake data UT for avro adapter](https://github.com/apache/arrow/issues/22830)
- [X] [[Java] Add support for skipping decoding of columns/field in Avro converter](https://github.com/apache/arrow/issues/22949)
- [ ] [[Java] Experiment with performance difference of avoiding the use of Avro Decoder](https://github.com/apache/arrow-java/issues/281)
- [X] [[Java] Support logical type encodings from Avro](https://github.com/apache/arrow/issues/22950)
- [ ] [[Java] Avro - Experiment with "compiled" consumer delegates for performance.](https://github.com/apache/arrow-java/issues/280)
- [X] [[JAVA] Avro adapter benchmark only runs once in JMH](https://github.com/apache/arrow/issues/23064)
- [X] [[Java] Extract a common base class for avro converter consumers](https://github.com/apache/arrow/issues/23236)
- [X] [[Java] Avro converter should convert attributes and props to FieldType metadata](https://github.com/apache/arrow/issues/23755)
**Note**: *This issue was originally created as [ARROW-5845](https://issues.apache.org/jira/browse/ARROW-5845). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Guide de contribution
Ouvrir le guide de contribution
Piste de recherche
Examinez les sous-tâches terminées pour comprendre l’adaptateur Avro existant, puis étudiez les deux expériences de performance restantes : éviter le Avro Decoder et utiliser des consumer delegates compilés. L’issue ne nomme aucun fichier d’implémentation, test ou critère d’acceptation pour ces expériences ; confirmez donc le périmètre prévu avec les maintainers avant de commencer.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- java
- Domaine
- data
- Type d'issue
- Fonctionnalité
- Difficulté
- 5/5
- Temps estimé
- Plus d'une semaine
- Activité
- À l'abandon
- Clarté
- À clarifier
- Accessibilité débutants
- 15/100