apache / apache/beam

[Bug]: Unable to use applyRowMutations() in bigquery IO apache beam java

Open
#27,616 10 comments 0 reactions 0 assignees View on GitHub
bug io java P2 stale
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What happened?

Create two tables in bigquery with primary key on table1 and table2 then create simple pipeline to read from table1 and upsert into table 2 with same schema

The primary key is will be a string with random unique value

```
PCollection rows1 = p.apply(BigQueryIO.readTableRows().from(" project.dataset.table1"));

PCollection rowmutate = rows1.apply(MapElements
.into(new TypeDescriptor() {
})
.via(tableRow -> RowMutation.of(tableRow,
RowMutationInformation.of(RowMutationInformation.MutationType.UPSERT, new Random().nextLong(1,1000000)))));

rowmutate.setCoder(RowMutation.RowMutationCoder.of())

.apply(BigQueryIO.applyRowMutations()
.to(" project.dataset.table2")
.withSchema(createSchema())
.withMethod(BigQueryIO.Write.Method.STORAGE_API_AT_LEAST_ONCE)
.withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_NEVER))
```
I am facing these two exceptions

`com.google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: The primary keys and clustering keys of the table are required to create an upsert stream. Entity: projects/myproject/datasets/mydataset/tables/mytable/streams/_default`

`Caused by: java.lang.RuntimeException: Append to stream projects/myproject/datasets/dataset/tables/table/streams/_default failed with stream doesn't exist
`

### Issue Priority

Priority: 1 (data loss / total loss of function)

### Issue Components

- [ ] Component: Python SDK
- [X] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [X] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.