honor conf "mapreduce.outputcommitter.factory.class" with PathOutputCommitterFactory in ParquetOutputFormat.getOutputCommitter
- Langage dominant
- Java
- Étoiles
- 3.1k
- Forks
- 1.6k
- Merge moyen
- 3 j 12 h
- PR mergées (30 j)
- 33
Description
The class ParquetOutputFormat extends org.apache.hadoop.mapreduce.lib.output.FileOutputFormat and override method getOutputCommitter(TaskAttemptContext), but unfortunatly, it replaces by a hard-coded new class instead of configurable factory PathOutputCommitterFactory that use configuration property "mapreduce.outputcommitter.factory.class"
See bugged overriding code :
```java
package org.apache.parquet.hadoop;
import org.apache.hadoop.mapreduce.OutputCommitter;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.PathOutputCommitter;
public class ParquetOutputFormat extends FileOutputFormat {
public synchronized OutputCommitter getOutputCommitter(TaskAttemptContext context) throws IOException {
if (this.committer == null) {
Path output = getOutputPath(context);
committer = new ParquetOutputCommitter(output, context); // <== BUG: hard coded class, ignore propoerty "mapreduce.outputcommitter.factory.class"
}
return this.committer;
}
```
See overriden code:
```java
package org.apache.hadoop.mapreduce.lib.output;
public abstract class FileOutputFormat extends OutputFormat {
@Override
public OutputCommitter getOutputCommitter(TaskAttemptContext context) throws IOException {
if (committer == null) {
Path output = getOutputPath(context);
committer = PathOutputCommitterFactory.getCommitterFactory(output, context.getConfiguration()).createOutputCommitter(output, context); // <== OK: configurable factory, accept property "mapreduce.outputcommitter.factory.class"
}
return committer;
}
```
**Reporter**: [Arnaud Nauwynck](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=arnaud.nauwynck)
#### Related issues:
- [performance problem in FileOutputCommitter for big list processed by single thread](https://issues.apache.org/jira/browse/MAPREDUCE-7465) (is related to)
#### PRs and other links:
- [GitHub Pull Request #1243](https://github.com/apache/parquet-mr/pull/1243)
- [GitHub Pull Request #1244](https://github.com/apache/parquet-mr/pull/1244)
**Note**: *This issue was originally created as [PARQUET-2416](https://issues.apache.org/jira/browse/PARQUET-2416). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
Guide de contribution
Aucun guide de contribution indexé pour ce dépôt
Piste de recherche
Commencez par org.apache.parquet.hadoop.ParquetOutputFormat.getOutputCommitter et comparez sa construction codée en dur de ParquetOutputCommitter avec FileOutputFormat.getOutputCommitter et PathOutputCommitterFactory. Examinez les pull requests #1243 et #1244 ; le travail est considéré comme terminé lorsque la configuration mapreduce.outputcommitter.factory.class est respectée sans perdre le comportement de commit de sortie de Parquet.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- java
- Domaine
- distributed-systems
- Type d'issue
- Bug
- Difficulté
- 3/5
- Temps estimé
- 1-2 jours
- Activité
- À l'abandon
- Clarté
- Plutôt claire
- Accessibilité débutants
- 25/100