googleapis / googleapis/google-cloud-java

[java-bigquerystorage] BigQueryWriteClient/JsonStreamWriter: missing values for duplicated, nested, proto-incompatible fields

Open
#11,968 1 comment 0 reactions 0 assignees View on GitHub
api: bigquerystorage priority: p3 type: bug
Dominant language
Java
Stars
2.1k
Forks
1.2k
Avg merge
1d 23h
Merged PRs (30d)
154

Description

#### Environment details

* OS type and version: macos Ventura 13.6
* Java version: 11
* version(s): 3.7.0, 2.47.0

#### Steps to reproduce

1. Create table in BigQuery:

```sql
CREATE TABLE test_dataset.test_table (
parent1 STRUCT< `1nested` STRING >,
parent2 STRUCT< `1nested` STRING >
)
```

So 2 parent `STRUCT` fields, both with a single field named `1nested`. `1` at the beginning is important here, which makes it proto incompatible based on this [class](https://github.com/googleapis/java-bigquerystorage/blob/54774f9bb6053b0130bd1a9a1533b1739e0b0263/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/BigQuerySchemaUtil.java#L33)

2. Write data to the table using `JsonStreamWriter`. Here is the [gist](https://gist.github.com/pondzix/bf5355dc82096593a72dc80bc41a87b7) with sample `scala-cli` script

The problem is: input contains values for both `parent1.1nested` and `parent2.1nested` but in BQ only the first one has correct value. The second one is null. Looks like it's lost somewhere:

![Screenshot 2024-07-23 at 09 32 49](https://github.com/user-attachments/assets/63deb937-0151-4b19-a2b4-c8c4d897e8a2)

#### Code example

From the gist ☝️:

```scala
//> using dep com.google.cloud:google-cloud-bigquerystorage:3.7.0
//> using dep com.google.cloud:google-cloud-bigquery:2.41.0

import com.google.cloud.bigquery.TableId
import com.google.cloud.bigquery.storage.v1.{BigQueryWriteClient, JsonStreamWriter}
import org.json.JSONArray
import scala.jdk.CollectionConverters._

/**
CREATE TABLE
test_dataset.test_table (
parent1 STRUCT< `1nested` STRING >,
parent2 STRUCT< `1nested` STRING >
)
*/

val input = Map(
"parent1" -> Map("1nested" -> "value").asJava, // "value" in BQ
"parent2" -> Map("1nested" -> "value").asJava, // null in BQ
)

write(input)

def write(input: Map[String, AnyRef]): Unit = {
val client = BigQueryWriteClient.create()
val streamId = TableId.of("...project..", "...dataset..", "..table..").getIAMResourceName + "/streams/_default"
val writer = JsonStreamWriter
.newBuilder(streamId, client)
.build

writer.append(new JSONArray(List(input.asJava).asJava)).get
writer.close()
client.close()
()
}
```

Any following attempt of writing data to `parent1.1nested` is successful, data is not null. Any following attempt of writing data to `parent2.1nested` always results in null in BQ.

In general, first proto-incompatible field "wins" and any other proto-incompatible field with the same name, living as a nested fields somewhere else, "lose".

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.