beam.io.BigQuerySink Python SDK does not support WRITE_TRUNCATE write disposition when specifying table with partition decorator
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
When trying to write to a BigQuery partition with this transform
pcoll | beam.io.Write(beam.io.BigQuerySink(table, schema=self.get_schema(),
create_disposition=beam.io.BigQueryDisposition.CREATE_NEVER,
write_disposition=beam.io.BigQueryDisposition.WRITE_TRUNCATE)))
I get the following error code:
"code": 400,
"message": "Invalid table ID \"table$20170103\". Table IDs must be alphanumeric (plus underscores) and must be at most 1024 characters long. Also, Table decorators cannot be used."
The table is partitioned by day already.
Imported from Jira [BEAM-2332](https://issues.apache.org/jira/browse/BEAM-2332). Original Jira may contain additional context.
Reported by: jc.rodrigue@gmail.com.
Contributor guide
Research direction
Start by tracing the Python SDK's beam.io.BigQuerySink handling of a partition-decorated table with WRITE_TRUNCATE, using the transform and error message in this issue to reproduce the failure. Done means the sink supports the shown partition table specification with WRITE_TRUNCATE without producing BigQuery's invalid table ID error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, python
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100