apache / apache/beam

Equivalent to withNumFileShards() in Python as it is in Java

Open
#19,865 0 comments 0 reactions 0 assignees View on GitHub
bug gcp io P3 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

We would like to have an equivalent of withNumFileShards() in Python as well

.apply("WriteInBigQueryLoad", BigQueryIO.writeTableRows().to(tableLowUrgency)
.withSchema(schema)
.withMethod(Method.FILE_LOADS)
.withTriggeringFrequency(Duration.standardMinutes(2))
.withNumFileShards(1)
.withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED)
.withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_APPEND))

transformed | 'Write' \>\> beam.io.WriteToBigQuery(
known_args.target_table,
schema=schema,
create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED,
write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND,
method=beam.io.WriteToBigQuery.Method.FILE_LOADS,
triggering_frequency=5
)

Imported from Jira [BEAM-8190](https://issues.apache.org/jira/browse/BEAM-8190). Original Jira may contain additional context.
Reported by: sdominique.

Contributor guide

Open the contributing guide

Research direction

Start by locating Python's beam.io.WriteToBigQuery entry point and the Java BigQueryIO.writeTableRows implementation of withNumFileShards(). Compare their existing file-load options and related tests. Done means the Python API exposes equivalent file-shard behavior and its coverage verifies the requested configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, python
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.