BigQuery Partitioned table creation/write fails when destination has partition decorator
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
Following is the code that writes to BigQuery:
```
BigQueryIO.writeTableRows()
.to(destination)
.withCreateDisposition(CREATE_IF_NEEDED)
.withWriteDisposition(WRITE_APPEND)
.withSchema(tableSchema)
.expand(tableRows);
```
Here's the destination's implementation:
```
public TableDestination apply(ValueInSingleWindow input) {
String partition = timestampExtractor.apply(input.getValue())
.toString(DateTimeFormat.forPattern("yyyyMMdd").withZoneUTC());
TableReference tableReference = new
TableReference();
tableReference.setDatasetId(dataset);
tableReference.setProjectId(projectId);
tableReference.setTableId(String.format("%s_%s", table, partition));
log.debug("Will write to BigQuery
table: %s", tableReference);
return new TableDestination(tableReference, null);
}
```
When the dataflow tries to write to this table, I see the following message:
```
"errors" : [ {
"domain" : "global",
"message" : "Cannot read partition information from a table
that is not partitioned: :.$19730522",
"reason" : "invalid"
} ]
```
So, it looks like it's not creating tables with partition in the first place?
Apache beam version : 2.2.0
Imported from Jira [BEAM-3501](https://issues.apache.org/jira/browse/BEAM-3501). Original Jira may contain additional context.
Reported by: darshanmehta2.
Contributor guide
Research direction
Start at BigQueryIO.writeTableRows and follow CREATE_IF_NEEDED handling for TableDestination values containing partition decorators. Reproduce the supplied partitioned-table destination and verify that creation and subsequent append succeed without the “not partitioned” error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, java
- Domain
- cloud, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100