GoogleCloudPlatform / GoogleCloudPlatform/cloud-profiler-python

segmentation fault under high load

Open
#142 9 comments 5 reactions 1 assignee Claimed by @aabmass View on GitHub
Dominant language
Python
Stars
32
Forks
35
PR merge metrics
No merged PRs in 30d

Description

We noticed that we get sporadic restarts in one of our apps, especially when under high load.

Our apps are running on gke:
`k8s version: 1.26.7
machine image: cos_containerd
`

python version: 3.11
fastapi & uvicorn

google-cloud-profiler version: `4.1.0`

I've enabled debug logging in profiler, but I don't get much more details except:

`
DEBUG:googlecloudprofiler.client:Starting to create profile
DEBUG:googlecloudprofiler.client:Successfully created a CPU profile
Segmentation fault
`

some additional logs:
`
kernel: [ 8685.460126] uvicorn[86674]: segfault at 48 ip 00007f78dc517b01 sp 00007f78dcff7020 error 4 in _profiler.cpython-311-x86_64-linux-gnu.so[7f78dc514000+a5000] likely on CPU 0 (core 0, socket 0)
kernel: [ 8685.477270] Code: ff ff 0f 1f 80 00 00 00 00 41 54 55 48 89 fd 53 48 85 f6 0f 84 a8 00 00 00 48 8b 46 38 48 8b 58 08 48 85 db 0f 84 8c 00 00 00 <80> 7b 45 01 74 19 48 8b 43 20 48 63 90 a8 00 00 00 48 8d 84 50 b8
`

have you faced similar issues before and is there a possible solution/workaround?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.