apache / apache/beam

Deprecate .withProjection from ParquetIO

Open
#20,770 2 comments 0 reactions 1 assignee Claimed by @nitinware View on GitHub
good first issue improvement io java P3 parquet
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

There are multiple issues wrong with the API of withProjection:

1. The current API requires an extra encoderSchema that is not needed when projecting data in Parquet. The simplest way to get this with the Parquet API is by passing the projectionSchema like this:
> {color:#000000}AvroReadSupport{color}.setAvroReadSchema({color:#871094}conf{color}, {color:#871094}projectionSchema{color});
> olor:#000000}AvroReadSupport{color}.setRequestedProjection({color:#871094}conf{color}, {color:#871094}projectionSchema{color});
> We can offer an alternative method `withProjection(Configuration conf, List fields)` so users don't have to build their own projection Schema, but historically we have let users to rely on the upstream connector API. If we follow this we can better document in ParquetIO how to project fields by relying in the Parquet APIs and avoid maintaining this extra code in the Beam side.

 

Imported from Jira [BEAM-11908](https://issues.apache.org/jira/browse/BEAM-11908). Original Jira may contain additional context.
Reported by: iemejia.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.