apache / apache/arrow-java

[JAVA][C++]Support Parquet Read and Write in Java

Aberta
#279 8 comentários 1 reação 0 responsáveis Ver no GitHub
Type: enhancement
Linguagem predominante
Java
Estrelas
94
Forks
152
Merge médio
3d 16h
PRs com merge (30d)
11

Descrição

We added a new java interface to support parquet read and write from hdfs or local file.

The purpose of this implementation is that when we loading and dumping parquet data in Java, we can only use rowBased put and get methods. Since arrow already has C++ implementation to load and dump parquet, so we wrapped those codes as Java APIs.

After test, we noticed in our workload, performance improved more than 2x comparing with rowBased load and dump. So we want to contribute codes to arrow.

since this is a total independent change, there is no codes change to current arrow codes. We added two folders as listed:  java/adapter/parquet and cpp/src/jni/parquet

**Reporter**: [Chendi.Xue](https://issues.apache.org/jira/browse/ARROW-6720)
#### Related issues:
- [[Java][Dataset] Implement Datasets Java API ](https://github.com/apache/arrow/issues/17055) (incorporates)
- [[Java][Dataset] Support writing to files within dataset scanner via JNI](https://github.com/apache/arrow/issues/27628) (incorporates)
#### PRs and other links:
- [GitHub Pull Request apache/arrow#5522](https://github.com/apache/arrow/pull/5522)
- [GitHub Pull Request apache/arrow#5717](https://github.com/apache/arrow/pull/5717)
- [GitHub Pull Request apache/arrow#5719](https://github.com/apache/arrow/pull/5719)

**Note**: *This issue was originally created as [ARROW-6720](https://issues.apache.org/jira/browse/ARROW-6720). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Guia de contribuição

Abrir o guia de contribuição

Direção de pesquisa

Comece revisando java/adapter/parquet e cpp/src/jni/parquet e, em seguida, compare os pull requests vinculados #5522, #5717 e #5719 com as issues relacionadas a datasets. Verifique as APIs Java para leitura e gravação de Parquet no HDFS e em arquivos locais, além dos respectivos testes; considera-se concluído quando o suporte necessário estiver presente sem duplicar o trabalho de datasets já incorporado.

Escrita pelo modelo de indexação a partir do texto da issue.

Avaliação

Stack de tecnologia
cpp, java
Domínio
backend, data
Tipo de issue
Funcionalidade
Dificuldade
4/5
Tempo estimado
3-5 dias
Status de atividade
Estagnada
Clareza
Razoavelmente clara
Facilidade para iniciantes
20/100

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.