googleapis / googleapis/google-cloud-java

[java-bigquerystorage] BigQueryWriteClient/JsonStreamWriter: missing values for duplicated, nested, proto-incompatible fields

Đang mở
#11,968 1 bình luận 0 reaction 0 người được giao Xem trên GitHub
api: bigquerystorage priority: p3 type: bug
Ngôn ngữ chính
Java
Star
2.1k
Fork
1.2k
Merge trung bình
1 ngày 23 giờ
Pull request đã merge (30 ngày)
154

Mô tả

#### Environment details

* OS type and version: macos Ventura 13.6
* Java version: 11
* version(s): 3.7.0, 2.47.0

#### Steps to reproduce

1. Create table in BigQuery:

```sql
CREATE TABLE test_dataset.test_table (
parent1 STRUCT< `1nested` STRING >,
parent2 STRUCT< `1nested` STRING >
)
```

So 2 parent `STRUCT` fields, both with a single field named `1nested`. `1` at the beginning is important here, which makes it proto incompatible based on this [class](https://github.com/googleapis/java-bigquerystorage/blob/54774f9bb6053b0130bd1a9a1533b1739e0b0263/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/BigQuerySchemaUtil.java#L33)

2. Write data to the table using `JsonStreamWriter`. Here is the [gist](https://gist.github.com/pondzix/bf5355dc82096593a72dc80bc41a87b7) with sample `scala-cli` script

The problem is: input contains values for both `parent1.1nested` and `parent2.1nested` but in BQ only the first one has correct value. The second one is null. Looks like it's lost somewhere:

![Screenshot 2024-07-23 at 09 32 49](https://github.com/user-attachments/assets/63deb937-0151-4b19-a2b4-c8c4d897e8a2)

#### Code example

From the gist ☝️:

```scala
//> using dep com.google.cloud:google-cloud-bigquerystorage:3.7.0
//> using dep com.google.cloud:google-cloud-bigquery:2.41.0

import com.google.cloud.bigquery.TableId
import com.google.cloud.bigquery.storage.v1.{BigQueryWriteClient, JsonStreamWriter}
import org.json.JSONArray
import scala.jdk.CollectionConverters._

/**
CREATE TABLE
test_dataset.test_table (
parent1 STRUCT< `1nested` STRING >,
parent2 STRUCT< `1nested` STRING >
)
*/

val input = Map(
"parent1" -> Map("1nested" -> "value").asJava, // "value" in BQ
"parent2" -> Map("1nested" -> "value").asJava, // null in BQ
)

write(input)

def write(input: Map[String, AnyRef]): Unit = {
val client = BigQueryWriteClient.create()
val streamId = TableId.of("...project..", "...dataset..", "..table..").getIAMResourceName + "/streams/_default"
val writer = JsonStreamWriter
.newBuilder(streamId, client)
.build

writer.append(new JSONArray(List(input.asJava).asJava)).get
writer.close()
client.close()
()
}
```

Any following attempt of writing data to `parent1.1nested` is successful, data is not null. Any following attempt of writing data to `parent2.1nested` always results in null in BQ.

In general, first proto-incompatible field "wins" and any other proto-incompatible field with the same name, living as a nested fields somewhere else, "lose".

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.