googleapis / googleapis/google-cloud-python

Bigquery Python Library: `AppendRows` bidi-streaming RPC sends empty `x-goog-request-params` routing header, causing intermittent `InvalidArgument: Cannot route on empty project id ''``

Ouverte
#16,650 2 commentaires 1 réaction 1 personne assignée Assignée à @chalmerlowe Voir sur GitHub
api: bigquery priority: p2 type: bug
Langage dominant
Python
Étoiles
5.4k
Forks
1.8k
Merge moyen
3 j 4 h
PR mergées (30 j)
122

Description

### Determine this is the right repository

- [x] I determined this is the correct repository in which to report this bug.

### Summary of the issue

**Context**

Using `BigQueryWriteClient.append_rows()` to write data via the Storage Write API default stream (`_default`), with multiple sequential calls through the same client. This is in a multiprocessing setup where each worker creates its own `BigQueryWriteClient()` and makes many `append_rows` calls.

**Expected Behavior:**

`append_rows` should populate the `x-goog-request-params` gRPC routing header with the `write_stream` resource name (which contains the project ID), similar to how other methods in the same client do it:

```python
# create_write_stream — sets routing correctly
metadata = ... + (gapic_v1.routing_header.to_grpc_metadata((("parent", request.parent),)),)
```

**Actual Behavior:**

`append_rows` sends an empty routing header because it's a bidi-streaming RPC and the request iterator hasn't been consumed when metadata is set:

```python
# client.py, append_rows() — all versions 2.26.0 through 2.37.0
metadata = tuple(metadata) + (gapic_v1.routing_header.to_grpc_metadata(()),)
```

This produces `x-goog-request-params: ''`. The first several calls may succeed via the gateway's fallback routing, but after a gRPC reconnection (idle timeout, load balancing), the gateway cannot determine the target project and returns:

```
google.api_core.exceptions.InvalidArgument: 400 Cannot route on empty project id ''
```

**Reproduction:**

```python
from google.cloud.bigquery_storage_v1 import BigQueryWriteClient
from google.api_core.gapic_v1 import routing_header
import inspect

# Verify empty routing in source
source = inspect.getsource(BigQueryWriteClient.append_rows)
assert "to_grpc_metadata(())" in source

# Verify it produces an empty header
assert routing_header.to_grpc_metadata(()) == ("x-goog-request-params", "")
```

**Workaround:**

Pass routing metadata explicitly:

```python
metadata = (routing_header.to_grpc_metadata((("write_stream", stream_name),)),)
response = client.append_rows(requests=iter([request]), metadata=metadata)
```

**Environment:**
- `google-cloud-bigquery-storage`: 2.26.0 through 2.37.0 (all affected)
- Python 3.9

### API client name and version

_No response_

### Reproduction steps: code

file: main.py
```python
def reproduce():
# complete code here
```

### Reproduction steps: supporting files

file: mydata.csv
```
alpha,1,3
beta,2,5
```

### Reproduction steps: actual results

file: output.txtmydata.csv
```
Calculated: foo
```

### Reproduction steps: expected results

file: output.txtmydata.csv
```
Calculated: bar
```

### OS & version + platform

_No response_

### Python environment

_No response_

### Python dependencies

_No response_

### Additional context

_No response_

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.