apache / apache/beam

BigQuery Partitioned table creation/write fails when destination has partition decorator

Open
#18,680 0 comments 0 reactions 0 assignees View on GitHub
bug gcp io java P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

Following is the code that writes to BigQuery: 
```

BigQueryIO.writeTableRows()
.to(destination)
.withCreateDisposition(CREATE_IF_NEEDED)
.withWriteDisposition(WRITE_APPEND)

.withSchema(tableSchema)
.expand(tableRows);
```

 

Here's the destination's implementation: 
```

public TableDestination apply(ValueInSingleWindow input) {
String partition = timestampExtractor.apply(input.getValue())

.toString(DateTimeFormat.forPattern("yyyyMMdd").withZoneUTC());
TableReference tableReference = new
TableReference();
tableReference.setDatasetId(dataset);
tableReference.setProjectId(projectId);

tableReference.setTableId(String.format("%s_%s", table, partition));
log.debug("Will write to BigQuery
table: %s", tableReference);
return new TableDestination(tableReference, null);
}
```

 

When the dataflow tries to write to this table, I see the following message:
```

"errors" : [ {
"domain" : "global",
"message" : "Cannot read partition information from a table
that is not partitioned: :.$19730522",
"reason" : "invalid"
} ]
```

So, it looks like it's not creating tables with partition in the first place? 

Apache beam version : 2.2.0

Imported from Jira [BEAM-3501](https://issues.apache.org/jira/browse/BEAM-3501). Original Jira may contain additional context.
Reported by: darshanmehta2.

Contributor guide

Open the contributing guide

Research direction

Start at BigQueryIO.writeTableRows and follow CREATE_IF_NEEDED handling for TableDestination values containing partition decorators. Reproduce the supplied partitioned-table destination and verify that creation and subsequent append succeed without the “not partitioned” error.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, java
Domain
cloud, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.