googleapis / googleapis/google-cloud-python

Bigquery Python Library: `AppendRows` bidi-streaming RPC sends empty `x-goog-request-params` routing header, causing intermittent `InvalidArgument: Cannot route on empty project id ''``

Đang mở
#16,650 2 bình luận 1 reaction 1 người được giao Được giao cho @chalmerlowe Xem trên GitHub
api: bigquery priority: p2 type: bug
Ngôn ngữ chính
Python
Star
5.4k
Fork
1.8k
Merge trung bình
3 ngày 4 giờ
Pull request đã merge (30 ngày)
122

Mô tả

### Determine this is the right repository

- [x] I determined this is the correct repository in which to report this bug.

### Summary of the issue

**Context**

Using `BigQueryWriteClient.append_rows()` to write data via the Storage Write API default stream (`_default`), with multiple sequential calls through the same client. This is in a multiprocessing setup where each worker creates its own `BigQueryWriteClient()` and makes many `append_rows` calls.

**Expected Behavior:**

`append_rows` should populate the `x-goog-request-params` gRPC routing header with the `write_stream` resource name (which contains the project ID), similar to how other methods in the same client do it:

```python
# create_write_stream — sets routing correctly
metadata = ... + (gapic_v1.routing_header.to_grpc_metadata((("parent", request.parent),)),)
```

**Actual Behavior:**

`append_rows` sends an empty routing header because it's a bidi-streaming RPC and the request iterator hasn't been consumed when metadata is set:

```python
# client.py, append_rows() — all versions 2.26.0 through 2.37.0
metadata = tuple(metadata) + (gapic_v1.routing_header.to_grpc_metadata(()),)
```

This produces `x-goog-request-params: ''`. The first several calls may succeed via the gateway's fallback routing, but after a gRPC reconnection (idle timeout, load balancing), the gateway cannot determine the target project and returns:

```
google.api_core.exceptions.InvalidArgument: 400 Cannot route on empty project id ''
```

**Reproduction:**

```python
from google.cloud.bigquery_storage_v1 import BigQueryWriteClient
from google.api_core.gapic_v1 import routing_header
import inspect

# Verify empty routing in source
source = inspect.getsource(BigQueryWriteClient.append_rows)
assert "to_grpc_metadata(())" in source

# Verify it produces an empty header
assert routing_header.to_grpc_metadata(()) == ("x-goog-request-params", "")
```

**Workaround:**

Pass routing metadata explicitly:

```python
metadata = (routing_header.to_grpc_metadata((("write_stream", stream_name),)),)
response = client.append_rows(requests=iter([request]), metadata=metadata)
```

**Environment:**
- `google-cloud-bigquery-storage`: 2.26.0 through 2.37.0 (all affected)
- Python 3.9

### API client name and version

_No response_

### Reproduction steps: code

file: main.py
```python
def reproduce():
# complete code here
```

### Reproduction steps: supporting files

file: mydata.csv
```
alpha,1,3
beta,2,5
```

### Reproduction steps: actual results

file: output.txtmydata.csv
```
Calculated: foo
```

### Reproduction steps: expected results

file: output.txtmydata.csv
```
Calculated: bar
```

### OS & version + platform

_No response_

### Python environment

_No response_

### Python dependencies

_No response_

### Additional context

_No response_

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.