googleapis / googleapis/google-cloud-python

Bigquery Python Library: `AppendRows` bidi-streaming RPC sends empty `x-goog-request-params` routing header, causing intermittent `InvalidArgument: Cannot route on empty project id ''``

Aperta
#16,650 2 commenti 1 reazione 1 assegnatario Assegnata a @chalmerlowe Vedi su GitHub
api: bigquery priority: p2 type: bug
Lingua principale
Python
Stelle
5.4k
Fork
1.8k
Merge medio
3g 4h
PR unite (30g)
122

Descrizione

### Determine this is the right repository

- [x] I determined this is the correct repository in which to report this bug.

### Summary of the issue

**Context**

Using `BigQueryWriteClient.append_rows()` to write data via the Storage Write API default stream (`_default`), with multiple sequential calls through the same client. This is in a multiprocessing setup where each worker creates its own `BigQueryWriteClient()` and makes many `append_rows` calls.

**Expected Behavior:**

`append_rows` should populate the `x-goog-request-params` gRPC routing header with the `write_stream` resource name (which contains the project ID), similar to how other methods in the same client do it:

```python
# create_write_stream — sets routing correctly
metadata = ... + (gapic_v1.routing_header.to_grpc_metadata((("parent", request.parent),)),)
```

**Actual Behavior:**

`append_rows` sends an empty routing header because it's a bidi-streaming RPC and the request iterator hasn't been consumed when metadata is set:

```python
# client.py, append_rows() — all versions 2.26.0 through 2.37.0
metadata = tuple(metadata) + (gapic_v1.routing_header.to_grpc_metadata(()),)
```

This produces `x-goog-request-params: ''`. The first several calls may succeed via the gateway's fallback routing, but after a gRPC reconnection (idle timeout, load balancing), the gateway cannot determine the target project and returns:

```
google.api_core.exceptions.InvalidArgument: 400 Cannot route on empty project id ''
```

**Reproduction:**

```python
from google.cloud.bigquery_storage_v1 import BigQueryWriteClient
from google.api_core.gapic_v1 import routing_header
import inspect

# Verify empty routing in source
source = inspect.getsource(BigQueryWriteClient.append_rows)
assert "to_grpc_metadata(())" in source

# Verify it produces an empty header
assert routing_header.to_grpc_metadata(()) == ("x-goog-request-params", "")
```

**Workaround:**

Pass routing metadata explicitly:

```python
metadata = (routing_header.to_grpc_metadata((("write_stream", stream_name),)),)
response = client.append_rows(requests=iter([request]), metadata=metadata)
```

**Environment:**
- `google-cloud-bigquery-storage`: 2.26.0 through 2.37.0 (all affected)
- Python 3.9

### API client name and version

_No response_

### Reproduction steps: code

file: main.py
```python
def reproduce():
# complete code here
```

### Reproduction steps: supporting files

file: mydata.csv
```
alpha,1,3
beta,2,5
```

### Reproduction steps: actual results

file: output.txtmydata.csv
```
Calculated: foo
```

### Reproduction steps: expected results

file: output.txtmydata.csv
```
Calculated: bar
```

### OS & version + platform

_No response_

### Python environment

_No response_

### Python dependencies

_No response_

### Additional context

_No response_

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.