GoogleCloudPlatform / GoogleCloudPlatform/cloud-profiler-python

segmentation fault under high load

Offen
#142 9 Kommentare 5 Reaktionen 1 zugewiesene Person Beansprucht von @aabmass Auf GitHub ansehen
Vorherrschende Sprache
Python
Sterne
32
Forks
35
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

We noticed that we get sporadic restarts in one of our apps, especially when under high load.

Our apps are running on gke:
`k8s version: 1.26.7
machine image: cos_containerd
`

python version: 3.11
fastapi & uvicorn

google-cloud-profiler version: `4.1.0`

I've enabled debug logging in profiler, but I don't get much more details except:

`
DEBUG:googlecloudprofiler.client:Starting to create profile
DEBUG:googlecloudprofiler.client:Successfully created a CPU profile
Segmentation fault
`

some additional logs:
`
kernel: [ 8685.460126] uvicorn[86674]: segfault at 48 ip 00007f78dc517b01 sp 00007f78dcff7020 error 4 in _profiler.cpython-311-x86_64-linux-gnu.so[7f78dc514000+a5000] likely on CPU 0 (core 0, socket 0)
kernel: [ 8685.477270] Code: ff ff 0f 1f 80 00 00 00 00 41 54 55 48 89 fd 53 48 85 f6 0f 84 a8 00 00 00 48 8b 46 38 48 8b 58 08 48 85 db 0f 84 8c 00 00 00 <80> 7b 45 01 74 19 48 8b 43 20 48 63 90 a8 00 00 00 48 8d 84 50 b8
`

have you faced similar issues before and is there a possible solution/workaround?

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.