googleapis / googleapis/google-cloud-python

Memory leak on each execute_query

Abierto
#15,283 10 comentarios 0 reacciones 1 asignado Reclamado por @jackdingilian Ver en GitHub
api: bigtable
Lenguaje dominante
Python
Estrellas
5.4k
Forks
1.8k
Merge medio
3 d 4 h
PR fusionados (30 d)
122

Descripción

We are running a fairly simple Python service in Kubernetes - it's a simple webserver (via aiohttp) with REST API and it interacts with Google BigTable (both through mutations and SQL queries). The service is handling about 600 requests/second currently. We noticed, that the memory usage of the service kept growing indefinitely (well, until it hit the pod memory limit). After much trial and error we narrowed it down to Google BigTable. The main indicator for us was that how fast the memory grows depends on the amount of requests handled. So it was either the webserver, or BigTable...

We are using both `mutate_row` and `execute_query` and we did an experiment in production, where we patched the service to be able to stop it from sending either `mutate_row` or `execute_query` requests at runtime without restarting the service. We observed, that when we disabled `mutate_row`, the memory usage kept growing, while when we disabled `execute_query`, the curve flattened almost immediately - once we re-enabled `execute_query`, the memory usage started growing again.

Eventually we arrived to the smallest possible reproducer, which is basically just calling `SELECT 1 FROM table WHERE _key = @key` in a loop - see the code below and the attached memory usage graph.

We have only one instance of `BigTableAsyncClient` and one instance of `TableAsync`, we are running latest version of `google-cloud-bigtable`.

#### Environment details

- OS type and version: Debian 12 docker image in GKE
- Python version: 3.11.11
- pip version: 24.3.1
- `google-cloud-bigtable` version: 2.32.0

#### Steps to reproduce

1. Run the attached example
2. Observe unbounded memory growth

#### Code example

```python
import asyncio
import os
import uuid
from typing import Any

from google.cloud.bigtable.data import BigtableDataClientAsync

try:
BIGTABLE_PROJECT = os.environ.get("BIGTABLE_PROJECT")
BIGTABLE_INSTANCE = os.environ.get("BIGTABLE_INSTANCE")
BIGTABLE_TABLE = os.environ.get("BIGTABLE_TABLE")
except KeyError as e:
raise RuntimeError(f"Missing environment variable: {e}") from e

async def run_query(client: BigtableDataClientAsync, key: str) -> bool:
result = await client.execute_query(
f"SELECT 1 FROM {BIGTABLE_TABLE} WHERE _key = @key",
instance_id=BIGTABLE_INSTANCE,
parameters={"key": key.encode()},
app_profile_id="default",
)

row = await anext(result, None)
if row:
# Make sure we exhaust the iterator, although there should only ever be a single result
async for _ in result:
pass
return True
else:
return False

async def loop(client: BigtableDataClientAsync) -> None:
# just keep querying the client with random keys
while True:
await run_query(client, uuid.uuid4().hex)

async def run() -> None:
client = BigtableDataClientAsync(project=BIGTABLE_PROJECT)
# 10 tasks in parallel
tasks = [asyncio.create_task(loop(client)) for _ in range(10)]
await asyncio.gather(*tasks)

def main() -> None:
asyncio.run(run())

if __name__ == "__main__":
main()
```

## Memory Usage

This is a graph from Kubernetes, showcasing the unbounded memory growth of the Python script over the past few days.

Image

We also tried to debug the problem via [memray](https://github.com/bloomberg/memray) with grpcio compiled with debug symbols. There's some indication that the memory might be leaking inside grpcio - but we think it's weird that it would only leak when calling `execute_query` and not when calling `mutate_row` - so we think it maybe is some due to some interaction between this library and grpcio - but that's all just pure speculations from our side.

[memray-flamegraph-netrep-20250825-231600.html.gz](https://github.com/user-attachments/files/22203088/memray-flamegraph-netrep-20250825-231600.html.gz)

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.