googleapis / googleapis/google-cloud-java

[java-bigquerystorage] JsonToProtoMessage rejects base64 strings for BYTES columns

Abierto
#13,980 0 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Java
Estrellas
2.1k
Forks
1.2k
Merge medio
1 d 23 h
PR fusionados (30 d)
154

Descripción

## Issue Details

`JsonToProtoMessage` cannot convert a `BYTES` column from a JSON document that encodes the value as
a base64 string — the encoding that both protobuf's canonical JSON mapping and BigQuery's own JSON
ingestion use. It accepts only a `ByteString` (which cannot come out of parsed JSON text) or a
`JSONArray` of integer byte values such as `[104, 105]`.

This makes `JsonStreamWriter` unable to accept an otherwise ordinary JSON document, and it fails per
record rather than at setup.

### Why this looks like a defect rather than a missing feature

Two independent conventions say a JSON `bytes` value is a base64 string:

1. **Protobuf's canonical JSON mapping** — — states
for `bytes`: "JSON value will be the data encoded as a string using standard base64 encoding with
paddings. Either standard or URL-safe base64 encoding with/without paddings are accepted."
`protobuf-java-util` implements exactly this in `JsonFormat.ParserImpl#parseBytes`.

2. **BigQuery's own JSON ingestion** — [Loading JSON data from Cloud Storage](https://docs.cloud.google.com/bigquery/docs/loading-data-cloud-storage-json)
states "Columns with BYTES types must be encoded as Base64" in its data type conversion section
(the [CSV page](https://docs.cloud.google.com/bigquery/docs/loading-data-cloud-storage-csv)
carries the same line).

So a JSON document that `bq load` ingests without complaint fails per record through the Storage
Write API's own JSON converter. Conversely, `[104, 105]` — the form the converter does accept — is
not documented anywhere I could find and is not something a producer emits.

### Where

`google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/JsonToProtoMessage.java`,
in two places — `case BYTES:` in `fillField` (scalar) and in `fillRepeatedField`. Both fall through
to the "wrong field type" error when the value is a `String`.

Note that proto `BYTES` also carries BigQuery `NUMERIC`/`BIGNUMERIC`, handled in a prelude before the
plain `BYTES` handling. So any fix has to be guarded on
`fieldSchema != null && fieldSchema.getType() == TableFieldSchema.Type.BYTES` — with the overload
that takes no `TableSchema`, the converter cannot tell `BYTES` from `NUMERIC`, and decoding
unconditionally would turn today's error on a `NUMERIC` column into a silently wrong value. The
reproducer below includes a `NUMERIC` case so that behaviour is pinned.

### On the alphabet

Protobuf accepts standard and URL-safe, with and without padding. `JsonFormat.ParserImpl#parseBytes`
already answers how to do that — try standard, fall back to URL-safe on `IllegalArgumentException`.

`java.util.Base64` and Guava's `BaseEncoding` are behaviourally identical for this purpose, so the
choice is free. Measured on the JDK below, Guava 33.5.0-jre, one run:

| decoder | `aGk=` | `aGk` (unpadded) | `-_8=` (URL-safe) | `+/8=` (standard) |
|---|---|---|---|---|
| `java.util.Base64.getDecoder()` | ok | ok | error | ok |
| `java.util.Base64.getUrlDecoder()` | ok | ok | ok | error |
| `BaseEncoding.base64()` | ok | ok | error | ok |
| `BaseEncoding.base64Url()` | ok | ok | ok | error |

Both tolerate missing padding, and the two alphabets are mutually exclusive on `+/` versus `-_`, so
the try-then-fall-back shape is required rather than cosmetic.

**The pull request uses `java.util.Base64`**, following `AGENTS.md` §6's preference for the standard
library over an existing dependency. If you would rather match `JsonFormat.ParserImpl#parseBytes`
and use Guava's `BaseEncoding`, say so and I will switch it — the behaviour is identical, so it is a
two-line change.

### The pull request

A pull request is ready and follows immediately, per CONTRIBUTING's "open an issue before doing
significant work".

It is **additive**: a `String` in a `BYTES` column is an error today, so no input that works now
changes behaviour.

It also fixes a second, independent problem in the same `case BYTES:` branch, which I have filed
separately as #13979 — `fillField` rejects a scalar `byte[]` while `fillRepeatedField` accepts
one. Both live in the same few lines, so one pull request seemed easier to review than two touching
the same code; **happy to split it in two if you would prefer** to take them independently.

Scope: the pull request changes `v1` only. `com.google.cloud.bigquery.storage.v1beta2` carries its
own copy of `JsonToProtoMessage` with the same branch — tell me if you want it mirrored there and I
will add it.

## Environment

- OS Type and Version: macOS 26.5.2
- Java Version and JDK Vendor: OpenJDK 17.0.19, Eclipse Temurin (17.0.19+10)
- (If using GraalVM) GraalVM Version: n/a

Not deployed on GCP — this reproduces entirely locally, since the failure is in client-side
conversion before any RPC is made.

## Dependencies

- Libraries-Bom: `com.google.cloud:libraries-bom:26.85.1`
- Client library: `com.google.cloud:google-cloud-bigquerystorage:3.30.0` (resolved by the BOM
above)
- `com.google.protobuf:protobuf-java:4.33.2`
- `com.google.guava:guava:33.5.0-jre`
- `org.json:json` (the `JSONObject` parameter type of `convertToProtoMessage`)

Gax and auth versions are whatever the BOM resolves; no RPC is made, so they are not involved in the
failure.

## Reproducer

Single file, no GCP credentials or resources needed — the failure is entirely in local conversion.
Run with `google-cloud-bigquerystorage` on the classpath.

```java
import com.google.cloud.bigquery.storage.v1.*;
import com.google.protobuf.Descriptors;
import org.json.JSONObject;

public class B64Probe {
static TableFieldSchema f(String n, TableFieldSchema.Type t, TableFieldSchema.Mode m) {
return TableFieldSchema.newBuilder().setName(n).setType(t).setMode(m).build();
}

public static void main(String[] args) throws Exception {
TableSchema schema =
TableSchema.newBuilder()
.addFields(f("blob", TableFieldSchema.Type.BYTES, TableFieldSchema.Mode.NULLABLE))
.addFields(f("blobs", TableFieldSchema.Type.BYTES, TableFieldSchema.Mode.REPEATED))
.addFields(f("num", TableFieldSchema.Type.NUMERIC, TableFieldSchema.Mode.NULLABLE))
.build();
Descriptors.Descriptor d =
BQTableSchemaToProtoDescriptor.convertBQTableSchemaToProtoDescriptor(schema);

probe(d, schema, "scalar base64", "{\"blob\":\"aGk=\"}");
probe(d, schema, "scalar jsonarray", "{\"blob\":[104,105]}");
probe(d, schema, "repeated base64", "{\"blobs\":[\"aGk=\"]}");
probe(d, schema, "repeated jsonarray", "{\"blobs\":[[104,105]]}");
probe(d, schema, "numeric string", "{\"num\":\"1.5\"}");
}

static void probe(Descriptors.Descriptor d, TableSchema s, String label, String json) {
try {
Object msg =
JsonToProtoMessage.INSTANCE.convertToProtoMessage(d, s, new JSONObject(json), false);
System.out.println("OK " + label + " :: " + msg.toString().replace("\n", " ").trim());
} catch (Exception e) {
System.out.println("FAIL " + label + " :: " + e.getClass().getName() + ": " + e.getMessage());
}
}
}
```

Steps:

1. Put `google-cloud-bigquerystorage:3.30.0` (and its transitive dependencies) on the classpath.
2. Run the class above.
3. Observe that the two base64 cases fail while the `JSONArray` cases succeed.

`"aGk="` is base64 for `hi`, which is what `[104, 105]` produces.

## Logs and Stack Trace

Output of the reproducer, verbatim:

```
FAIL scalar base64 :: com.google.cloud.bigquery.storage.v1.Exceptions$RowIndexToErrorException: The map of row index to error message is {0=Field root.blob failed to convert to BYTES. Error: JSONObject does not have a bytes field at root.blob.}
OK scalar jsonarray :: blob: "hi"
FAIL repeated base64 :: com.google.cloud.bigquery.storage.v1.Exceptions$RowIndexToErrorException: The map of row index to error message is {0=Field root.blobs failed to convert to BYTES. Error: JSONObject does not have a bytes field at root.blobs[0].}
OK repeated jsonarray :: blobs: "hi"
OK numeric string :: num: "\000/hY"
```

The last line is the `NUMERIC` case that a fix must not disturb.

## Behavior

- **When did the issue begin?** Not a regression. The branch already had this
`ByteString`/`JSONArray` shape in April 2021 — see
[googleapis/java-bigquerystorage#984](https://github.com/googleapis/java-bigquerystorage/pull/984).
- **Is it flaky?** No, fully deterministic and reproducible offline.
- **Related to volume of data?** No.

Guía de contribución

Abrir la guía de contribución

Línea de trabajo

Start in google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1/JsonToProtoMessage.java, at the BYTES branches in fillField and fillRepeatedField. Run the self-contained B64Probe reproducer to verify scalar and repeated base64 strings, while confirming the NUMERIC string case remains unchanged; the issue notes that a pull request is already ready.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
google-cloud, java, json
Área
backend, cloud, data-engineering
Tipo de issue
Error
Dificultad
2/5
Tiempo estimado
1-3 horas
Estado de actividad
Estancado
Claridad
Bien especificado
Aptitud para principiantes
20/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.