googleapis / googleapis/google-cloud-java
[java-bigquerystorage] BigQueryWriteClient/JsonStreamWriter: missing values for duplicated, nested, proto-incompatible fields
- Ngôn ngữ chính
- Java
- Star
- 2.1k
- Fork
- 1.2k
- Merge trung bình
- 1 ngày 23 giờ
- Pull request đã merge (30 ngày)
- 154
Mô tả
#### Environment details
* OS type and version: macos Ventura 13.6
* Java version: 11
* version(s): 3.7.0, 2.47.0
#### Steps to reproduce
1. Create table in BigQuery:
```sql
CREATE TABLE test_dataset.test_table (
parent1 STRUCT< `1nested` STRING >,
parent2 STRUCT< `1nested` STRING >
)
```
So 2 parent `STRUCT` fields, both with a single field named `1nested`. `1` at the beginning is important here, which makes it proto incompatible based on this [class](https://github.com/googleapis/java-bigquerystorage/blob/54774f9bb6053b0130bd1a9a1533b1739e0b0263/google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/BigQuerySchemaUtil.java#L33)
2. Write data to the table using `JsonStreamWriter`. Here is the [gist](https://gist.github.com/pondzix/bf5355dc82096593a72dc80bc41a87b7) with sample `scala-cli` script
The problem is: input contains values for both `parent1.1nested` and `parent2.1nested` but in BQ only the first one has correct value. The second one is null. Looks like it's lost somewhere:

#### Code example
From the gist ☝️:
```scala
//> using dep com.google.cloud:google-cloud-bigquerystorage:3.7.0
//> using dep com.google.cloud:google-cloud-bigquery:2.41.0
import com.google.cloud.bigquery.TableId
import com.google.cloud.bigquery.storage.v1.{BigQueryWriteClient, JsonStreamWriter}
import org.json.JSONArray
import scala.jdk.CollectionConverters._
/**
CREATE TABLE
test_dataset.test_table (
parent1 STRUCT< `1nested` STRING >,
parent2 STRUCT< `1nested` STRING >
)
*/
val input = Map(
"parent1" -> Map("1nested" -> "value").asJava, // "value" in BQ
"parent2" -> Map("1nested" -> "value").asJava, // null in BQ
)
write(input)
def write(input: Map[String, AnyRef]): Unit = {
val client = BigQueryWriteClient.create()
val streamId = TableId.of("...project..", "...dataset..", "..table..").getIAMResourceName + "/streams/_default"
val writer = JsonStreamWriter
.newBuilder(streamId, client)
.build
writer.append(new JSONArray(List(input.asJava).asJava)).get
writer.close()
client.close()
()
}
```
Any following attempt of writing data to `parent1.1nested` is successful, data is not null. Any following attempt of writing data to `parent2.1nested` always results in null in BQ.
In general, first proto-incompatible field "wins" and any other proto-incompatible field with the same name, living as a nested fields somewhere else, "lose".
Hướng dẫn đóng góp
Đánh giá
Issue này chưa được đánh giá.