googleapis / googleapis/google-cloud-python

Memory leak on each execute_query

オープン
#15,283 コメント 10 件 リアクション 0 件 担当者 1 名 @jackdingilian が担当を希望しています GitHub で見る
api: bigtable
主要言語
Python
スター
5.4k
フォーク
1.8k
平均マージ
3日 4時間
マージ済み PR(30日)
122

説明

We are running a fairly simple Python service in Kubernetes - it's a simple webserver (via aiohttp) with REST API and it interacts with Google BigTable (both through mutations and SQL queries). The service is handling about 600 requests/second currently. We noticed, that the memory usage of the service kept growing indefinitely (well, until it hit the pod memory limit). After much trial and error we narrowed it down to Google BigTable. The main indicator for us was that how fast the memory grows depends on the amount of requests handled. So it was either the webserver, or BigTable...

We are using both `mutate_row` and `execute_query` and we did an experiment in production, where we patched the service to be able to stop it from sending either `mutate_row` or `execute_query` requests at runtime without restarting the service. We observed, that when we disabled `mutate_row`, the memory usage kept growing, while when we disabled `execute_query`, the curve flattened almost immediately - once we re-enabled `execute_query`, the memory usage started growing again.

Eventually we arrived to the smallest possible reproducer, which is basically just calling `SELECT 1 FROM table WHERE _key = @key` in a loop - see the code below and the attached memory usage graph.

We have only one instance of `BigTableAsyncClient` and one instance of `TableAsync`, we are running latest version of `google-cloud-bigtable`.

#### Environment details

- OS type and version: Debian 12 docker image in GKE
- Python version: 3.11.11
- pip version: 24.3.1
- `google-cloud-bigtable` version: 2.32.0

#### Steps to reproduce

1. Run the attached example
2. Observe unbounded memory growth

#### Code example

```python
import asyncio
import os
import uuid
from typing import Any

from google.cloud.bigtable.data import BigtableDataClientAsync

try:
BIGTABLE_PROJECT = os.environ.get("BIGTABLE_PROJECT")
BIGTABLE_INSTANCE = os.environ.get("BIGTABLE_INSTANCE")
BIGTABLE_TABLE = os.environ.get("BIGTABLE_TABLE")
except KeyError as e:
raise RuntimeError(f"Missing environment variable: {e}") from e

async def run_query(client: BigtableDataClientAsync, key: str) -> bool:
result = await client.execute_query(
f"SELECT 1 FROM {BIGTABLE_TABLE} WHERE _key = @key",
instance_id=BIGTABLE_INSTANCE,
parameters={"key": key.encode()},
app_profile_id="default",
)

row = await anext(result, None)
if row:
# Make sure we exhaust the iterator, although there should only ever be a single result
async for _ in result:
pass
return True
else:
return False

async def loop(client: BigtableDataClientAsync) -> None:
# just keep querying the client with random keys
while True:
await run_query(client, uuid.uuid4().hex)

async def run() -> None:
client = BigtableDataClientAsync(project=BIGTABLE_PROJECT)
# 10 tasks in parallel
tasks = [asyncio.create_task(loop(client)) for _ in range(10)]
await asyncio.gather(*tasks)

def main() -> None:
asyncio.run(run())

if __name__ == "__main__":
main()
```

## Memory Usage

This is a graph from Kubernetes, showcasing the unbounded memory growth of the Python script over the past few days.

Image

We also tried to debug the problem via [memray](https://github.com/bloomberg/memray) with grpcio compiled with debug symbols. There's some indication that the memory might be leaking inside grpcio - but we think it's weird that it would only leak when calling `execute_query` and not when calling `mutate_row` - so we think it maybe is some due to some interaction between this library and grpcio - but that's all just pure speculations from our side.

[memray-flamegraph-netrep-20250825-231600.html.gz](https://github.com/user-attachments/files/22203088/memray-flamegraph-netrep-20250825-231600.html.gz)

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。