apache / apache/beam

setting numshards dynamically for TextIO.write()

Open
#19,354 0 comments 0 reactions 0 assignees View on GitHub
improvement io java P3 text
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

I go through code for TextIO. It uses WriteFiles  in TypedWrite. Now, WriteFiles has the functionality withNumShards(ValueProvider) and [withSharding](https://beam.apache.org/releases/javadoc/2.9.0/org/apache/beam/sdk/io/WriteFiles.html#withSharding-org.apache.beam.sdk.transforms.PTransform-)([PTransform](https://beam.apache.org/releases/javadoc/2.9.0/org/apache/beam/sdk/transforms/PTransform.html)<[PCollection](https://beam.apache.org/releases/javadoc/2.9.0/org/apache/beam/sdk/values/PCollection.html)<[UserT](https://beam.apache.org/releases/javadoc/2.9.0/org/apache/beam/sdk/io/WriteFiles.html)\>,[PCollectionView](https://beam.apache.org/releases/javadoc/2.9.0/org/apache/beam/sdk/values/PCollectionView.html)\> sharding) but TextIO or its inner class TypedWrite doesn't have that. It would really be a great to expose that functionality to user.

Are there some issues in implementation that this is not already available?

Imported from Jira [BEAM-6721](https://issues.apache.org/jira/browse/BEAM-6721). Original Jira may contain additional context.
Reported by: aditya.guru149.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.