apache / apache/beam

[Task]: Implement FileIO SchemaTransform Provider

Open
#22,614 1 comment 0 reactions 0 assignees View on GitHub
files io new feature P2
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What needs to happen?

The output of this task is a [TypedSchemaTransformProvider](https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/TypedSchemaTransformProvider.java) extension for reading from and writing files using the FileIO.Read and FileIO.Write transforms, respectively.

Briefly described, a Read and Write configuration informs a [TypedSchemaTransformProvider](https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/TypedSchemaTransformProvider.java) extension for details on how to build a corresponding FileIO.Read and FileIO.Write [SchemaTransform](https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/SchemaTransform.java).

See [BigQuerySchemaTransformReadProvider](https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQuerySchemaTransformReadProvider.java) and [BigQuerySchemaTransformWriteProvider](https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQuerySchemaTransformWriteProvider.java) for an example design pattern.

### Issue Priority

Priority: 1

### Issue Component

Component: io-ideas

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.