googleapis / googleapis/google-cloud-python

Memory leak on each execute_query

Offen
#15,283 10 Kommentare 0 Reaktionen 1 zugewiesene Person Beansprucht von @jackdingilian Auf GitHub ansehen
api: bigtable
Vorherrschende Sprache
Python
Sterne
5.4k
Forks
1.8k
Ø Merge
3 T. 4 Std.
Gemergte PRs (30 T.)
122

Beschreibung

We are running a fairly simple Python service in Kubernetes - it's a simple webserver (via aiohttp) with REST API and it interacts with Google BigTable (both through mutations and SQL queries). The service is handling about 600 requests/second currently. We noticed, that the memory usage of the service kept growing indefinitely (well, until it hit the pod memory limit). After much trial and error we narrowed it down to Google BigTable. The main indicator for us was that how fast the memory grows depends on the amount of requests handled. So it was either the webserver, or BigTable...

We are using both `mutate_row` and `execute_query` and we did an experiment in production, where we patched the service to be able to stop it from sending either `mutate_row` or `execute_query` requests at runtime without restarting the service. We observed, that when we disabled `mutate_row`, the memory usage kept growing, while when we disabled `execute_query`, the curve flattened almost immediately - once we re-enabled `execute_query`, the memory usage started growing again.

Eventually we arrived to the smallest possible reproducer, which is basically just calling `SELECT 1 FROM table WHERE _key = @key` in a loop - see the code below and the attached memory usage graph.

We have only one instance of `BigTableAsyncClient` and one instance of `TableAsync`, we are running latest version of `google-cloud-bigtable`.

#### Environment details

- OS type and version: Debian 12 docker image in GKE
- Python version: 3.11.11
- pip version: 24.3.1
- `google-cloud-bigtable` version: 2.32.0

#### Steps to reproduce

1. Run the attached example
2. Observe unbounded memory growth

#### Code example

```python
import asyncio
import os
import uuid
from typing import Any

from google.cloud.bigtable.data import BigtableDataClientAsync

try:
BIGTABLE_PROJECT = os.environ.get("BIGTABLE_PROJECT")
BIGTABLE_INSTANCE = os.environ.get("BIGTABLE_INSTANCE")
BIGTABLE_TABLE = os.environ.get("BIGTABLE_TABLE")
except KeyError as e:
raise RuntimeError(f"Missing environment variable: {e}") from e

async def run_query(client: BigtableDataClientAsync, key: str) -> bool:
result = await client.execute_query(
f"SELECT 1 FROM {BIGTABLE_TABLE} WHERE _key = @key",
instance_id=BIGTABLE_INSTANCE,
parameters={"key": key.encode()},
app_profile_id="default",
)

row = await anext(result, None)
if row:
# Make sure we exhaust the iterator, although there should only ever be a single result
async for _ in result:
pass
return True
else:
return False

async def loop(client: BigtableDataClientAsync) -> None:
# just keep querying the client with random keys
while True:
await run_query(client, uuid.uuid4().hex)

async def run() -> None:
client = BigtableDataClientAsync(project=BIGTABLE_PROJECT)
# 10 tasks in parallel
tasks = [asyncio.create_task(loop(client)) for _ in range(10)]
await asyncio.gather(*tasks)

def main() -> None:
asyncio.run(run())

if __name__ == "__main__":
main()
```

## Memory Usage

This is a graph from Kubernetes, showcasing the unbounded memory growth of the Python script over the past few days.

Image

We also tried to debug the problem via [memray](https://github.com/bloomberg/memray) with grpcio compiled with debug symbols. There's some indication that the memory might be leaking inside grpcio - but we think it's weird that it would only leak when calling `execute_query` and not when calling `mutate_row` - so we think it maybe is some due to some interaction between this library and grpcio - but that's all just pure speculations from our side.

[memray-flamegraph-netrep-20250825-231600.html.gz](https://github.com/user-attachments/files/22203088/memray-flamegraph-netrep-20250825-231600.html.gz)

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.