googleapis / googleapis/google-cloud-python

Bigquery Python Library: `AppendRows` bidi-streaming RPC sends empty `x-goog-request-params` routing header, causing intermittent `InvalidArgument: Cannot route on empty project id ''``

Abierto
#16,650 2 comentarios 1 reacción 1 asignado Asignado a @chalmerlowe Ver en GitHub
api: bigquery priority: p2 type: bug
Lenguaje dominante
Python
Estrellas
5.4k
Forks
1.8k
Merge medio
3 d 4 h
PR fusionados (30 d)
122

Descripción

### Determine this is the right repository

- [x] I determined this is the correct repository in which to report this bug.

### Summary of the issue

**Context**

Using `BigQueryWriteClient.append_rows()` to write data via the Storage Write API default stream (`_default`), with multiple sequential calls through the same client. This is in a multiprocessing setup where each worker creates its own `BigQueryWriteClient()` and makes many `append_rows` calls.

**Expected Behavior:**

`append_rows` should populate the `x-goog-request-params` gRPC routing header with the `write_stream` resource name (which contains the project ID), similar to how other methods in the same client do it:

```python
# create_write_stream — sets routing correctly
metadata = ... + (gapic_v1.routing_header.to_grpc_metadata((("parent", request.parent),)),)
```

**Actual Behavior:**

`append_rows` sends an empty routing header because it's a bidi-streaming RPC and the request iterator hasn't been consumed when metadata is set:

```python
# client.py, append_rows() — all versions 2.26.0 through 2.37.0
metadata = tuple(metadata) + (gapic_v1.routing_header.to_grpc_metadata(()),)
```

This produces `x-goog-request-params: ''`. The first several calls may succeed via the gateway's fallback routing, but after a gRPC reconnection (idle timeout, load balancing), the gateway cannot determine the target project and returns:

```
google.api_core.exceptions.InvalidArgument: 400 Cannot route on empty project id ''
```

**Reproduction:**

```python
from google.cloud.bigquery_storage_v1 import BigQueryWriteClient
from google.api_core.gapic_v1 import routing_header
import inspect

# Verify empty routing in source
source = inspect.getsource(BigQueryWriteClient.append_rows)
assert "to_grpc_metadata(())" in source

# Verify it produces an empty header
assert routing_header.to_grpc_metadata(()) == ("x-goog-request-params", "")
```

**Workaround:**

Pass routing metadata explicitly:

```python
metadata = (routing_header.to_grpc_metadata((("write_stream", stream_name),)),)
response = client.append_rows(requests=iter([request]), metadata=metadata)
```

**Environment:**
- `google-cloud-bigquery-storage`: 2.26.0 through 2.37.0 (all affected)
- Python 3.9

### API client name and version

_No response_

### Reproduction steps: code

file: main.py
```python
def reproduce():
# complete code here
```

### Reproduction steps: supporting files

file: mydata.csv
```
alpha,1,3
beta,2,5
```

### Reproduction steps: actual results

file: output.txtmydata.csv
```
Calculated: foo
```

### Reproduction steps: expected results

file: output.txtmydata.csv
```
Calculated: bar
```

### OS & version + platform

_No response_

### Python environment

_No response_

### Python dependencies

_No response_

### Additional context

_No response_

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.