HadoopInputFormatIO should support custom key/value coder
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
Right now `HadoopInputFormatIO` infers coders with `getDefaultCoder` but this doesn't work for cases like `AvroCoder` with `GenericRecord`, for example, when Avro is used with Parquet.
https://github.com/apache/beam/blob/master/sdks/java/io/hadoop/input-format/src/main/java/org/apache/beam/sdk/io/hadoop/inputformat/HadoopInputFormatIO.java#L288
Imported from Jira [BEAM-2765](https://issues.apache.org/jira/browse/BEAM-2765). Original Jira may contain additional context.
Reported by: sinisa_lyh.
Contributor guide
Research direction
Start in sdks/java/io/hadoop/input-format/src/main/java/org/apache/beam/sdk/io/hadoop/inputformat/HadoopInputFormatIO.java around line 288 and read how getDefaultCoder infers key and value coders. Trace the existing Hadoop input configuration, then define completion as allowing callers to provide custom key/value coders so AvroCoder with GenericRecord works for Avro-backed formats such as Parquet.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100