apache / apache/beam

beam.io.BigQuerySink Python SDK does not support WRITE_TRUNCATE write disposition when specifying table with partition decorator

Open
#18,267 1 comment 0 reactions 0 assignees View on GitHub
bug core P3 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

When trying to write to a BigQuery partition with this transform

pcoll | beam.io.Write(beam.io.BigQuerySink(table, schema=self.get_schema(),
create_disposition=beam.io.BigQueryDisposition.CREATE_NEVER,
write_disposition=beam.io.BigQueryDisposition.WRITE_TRUNCATE)))

I get the following error code:

"code": 400,
"message": "Invalid table ID \"table$20170103\". Table IDs must be alphanumeric (plus underscores) and must be at most 1024 characters long. Also, Table decorators cannot be used."

The table is partitioned by day already.

Imported from Jira [BEAM-2332](https://issues.apache.org/jira/browse/BEAM-2332). Original Jira may contain additional context.
Reported by: jc.rodrigue@gmail.com.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the Python SDK's beam.io.BigQuerySink handling of a partition-decorated table with WRITE_TRUNCATE, using the transform and error message in this issue to reproduce the failure. Done means the sink supports the shown partition table specification with WRITE_TRUNCATE without producing BigQuery's invalid table ID error.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, python
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.