apache / apache/parquet-java

Parquet Combine Input format for MapReduce job

Open
#2,070 0 comments 0 reactions 0 assignees View on GitHub
Component: Java Component: Parquet Priority: Major Type: enhancement
Dominant language
Java
Stars
3.1k
Forks
1.6k
Avg merge
3d 12h
Merged PRs (30d)
33

Description

ParquetInputFormat can only process one parquet file, if there are small files, it will spawn many small map task processing the small file. It is not efficient.
We need to provide CombineInputFormat to combine small files together in one map task.

**Reporter**: [Bing Jiang](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=jiangbinglover)

**Note**: *This issue was originally created as [PARQUET-1051](https://issues.apache.org/jira/browse/PARQUET-1051). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.