grafana / grafana/pyroscope-dotnet

Pyroscope profiling doesn't work on aks pods

Open
#401 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
38
Forks
13
Avg merge
20h 15m
Merged PRs (30d)
10

Description

Hi,

I am having a great problem with getting Pyroscope profiling to actually work in my aks pods.
I have successfully run the app image locally via `podman run` and it was working and sending the data to port-forwarded pyroscope in a proper way.
However, the same image, with the same exact envs is not producing profiles at all.

When starting the local container I see
```
[2026-08-08 14:08:33.366 | debug | PId: 1 | TId: 292] ManagedThreadList::SetThreadName(clrThreadId: 0x7f4b58010500, pThreadName: ".NET TP Worker") completed for ProfilerThreadInfoId=25.
[2026-08-08 14:08:33.368 | debug | PId: 1 | TId: 293] Callback invoked: ThreadCreated(threadId=0x7f4b58010500)
[2026-08-08 14:08:33.368 | debug | PId: 1 | TId: 293] Callback invoked: ThreadAssignedToOSThread(managedThreadId=0x7f4b58010500, osThreadId=293)
[2026-08-08 14:08:33.368 | debug | PId: 1 | TId: 293] ManagedThreadList::SetThreadOsInfo(clrThreadId: 0x7f4b58010500, osThreadId: 293, osThreadHandle: 0x0xab8) completed for ProfilerThreadInfoId=37.
[2026-08-08 14:08:33.368 | debug | PId: 1 | TId: 293] Creating timer for thread 293
[2026-08-08 14:08:33.368 | debug | PId: 1 | TId: 293] CorProfilerCallback::ThreadNameChanged(threadId=0x7f4b9818e180, name=".NET TP Worker")
[2026-08-08 14:08:33.368 | debug | PId: 1 | TId: 293] Map (0) to .NET TP Worker
[2026-08-08 14:08:33.368 | debug | PId: 1 | TId: 293] ManagedThreadList::SetThreadName(clrThreadId: 0x7f4b9818e180, pThreadName: ".NET TP Worker") completed for ProfilerThreadInfoId=26.
[2026-08-08 14:08:33.369 | debug | PId: 1 | TId: 294] Callback invoked: ThreadCreated(threadId=0x7f4b9818e180)
[2026-08-08 14:08:33.369 | debug | PId: 1 | TId: 294] Callback invoked: ThreadAssignedToOSThread(managedThreadId=0x7f4b9818e180, osThreadId=294)
[2026-08-08 14:08:33.369 | debug | PId: 1 | TId: 294] ManagedThreadList::SetThreadOsInfo(clrThreadId: 0x7f4b9818e180, osThreadId: 294, osThreadHandle: 0x0xacc) completed for ProfilerThreadInfoId=38.
[2026-08-08 14:08:33.369 | debug | PId: 1 | TId: 294] Creating timer for thread 294
[2026-08-08 14:08:33.369 | debug | PId: 1 | TId: 294] CorProfilerCallback::ThreadNameChanged(threadId=0x7f4b5c002a10, name=".NET TP Worker")
[2026-08-08 14:08:33.369 | debug | PId: 1 | TId: 294] Map (0) to .NET TP Worker
[2026-08-08 14:08:33.369 | debug | PId: 1 | TId: 294] ManagedThreadList::SetThreadName(clrThreadId: 0x7f4b5c002a10, pThreadName: ".NET TP Worker") completed for ProfilerThreadInfoId=27.
[2026-08-08 14:08:33.372 | debug | PId: 1 | TId: 295] Callback invoked: ThreadCreated(threadId=0x7f4b5c002a10)
[2026-08-08 14:08:33.372 | debug | PId: 1 | TId: 295] Callback invoked: ThreadAssignedToOSThread(managedThreadId=0x7f4b5c002a10, osThreadId=295)
[2026-08-08 14:08:33.372 | debug | PId: 1 | TId: 295] ManagedThreadList::SetThreadOsInfo(clrThreadId: 0x7f4b5c002a10, osThreadId: 295, osThreadHandle: 0x0xad8) completed for ProfilerThreadInfoId=39.
[2026-08-08 14:08:33.372 | debug | PId: 1 | TId: 295] Creating timer for thread 295
[2026-08-08 14:08:33.372 | debug | PId: 1 | TId: 295] CorProfilerCallback::ThreadNameChanged(threadId=0x7f4b50002a10, name=".NET TP Worker")
[2026-08-08 14:08:33.372 | debug | PId: 1 | TId: 295] Map (0) to .NET TP Worker
[2026-08-08 14:08:33.372 | debug | PId: 1 | TId: 295] ManagedThreadList::SetThreadName(clrThreadId: 0x7f4b50002a10, pThreadName: ".NET TP Worker") completed for ProfilerThreadInfoId=28.
[2026-08-08 14:08:33.373 | debug | PId: 1 | TId: 296] Callback invoked: ThreadCreated(threadId=0x7f4b50002a10)
[2026-08-08 14:08:33.373 | debug | PId: 1 | TId: 296] Callback invoked: ThreadAssignedToOSThread(managedThreadId=0x7f4b50002a10, osThreadId=296)
[2026-08-08 14:08:33.373 | debug | PId: 1 | TId: 296] ManagedThreadList::SetThreadOsInfo(clrThreadId: 0x7f4b50002a10, osThreadId: 296, osThreadHandle: 0x0xae4) completed for ProfilerThreadInfoId=40.
[2026-08-08 14:08:33.373 | debug | PId: 1 | TId: 296] Creating timer for thread 296
[2026-08-08 14:08:33.373 | debug | PId: 1 | TId: 296] CorProfilerCallback::ThreadNameChanged(threadId=0x7f4b54002a10, name=".NET TP Worker")
[2026-08-08 14:08:33.373 | debug | PId: 1 | TId: 296] Map (0) to .NET TP Worker
[2026-08-08 14:08:33.373 | debug | PId: 1 | TId: 296] ManagedThreadList::SetThreadName(clrThreadId: 0x7f4b54002a10, pThreadName: ".NET TP Worker") completed for ProfilerThreadInfoId=29.
[2026-08-08 14:08:33.374 | debug | PId: 1 | TId: 297] Callback invoked: ThreadCreated(threadId=0x7f4b54002a10)
[2026-08-08 14:08:33.374 | debug | PId: 1 | TId: 297] Callback invoked: ThreadAssignedToOSThread(managedThreadId=0x7f4b54002a10, osThreadId=297)
[2026-08-08 14:08:33.374 | debug | PId: 1 | TId: 297] ManagedThreadList::SetThreadOsInfo(clrThreadId: 0x7f4b54002a10, osThreadId: 297, osThreadHandle: 0x0xaf0) completed for ProfilerThreadInfoId=41.
[2026-08-08 14:08:33.374 | debug | PId: 1 | TId: 297] Creating timer for thread 297
[2026-08-08 14:08:33.374 | debug | PId: 1 | TId: 297] CorProfilerCallback::ThreadNameChanged(threadId=0x7f4b48002a10, name=".NET TP Worker")
[2026-08-08 14:08:33.374 | debug | PId: 1 | TId: 297] Map (0) to .NET TP Worker
[2026-08-08 14:08:33.374 | debug | PId: 1 | TId: 297] ManagedThreadList::SetThreadName(clrThreadId: 0x7f4b48002a10, pThreadName: ".NET TP Worker") completed for ProfilerThreadInfoId=2a.
[2026-08-08 14:08:33.378 | debug | PId: 1 | TId: 298] Callback invoked: ThreadCreated(threadId=0x7f4b48002a10)
[2026-08-08 14:08:33.378 | debug | PId: 1 | TId: 298] Callback invoked: ThreadAssignedToOSThread(managedThreadId=0x7f4b48002a10, osThreadId=298)
[2026-08-08 14:08:33.378 | debug | PId: 1 | TId: 298] ManagedThreadList::SetThreadOsInfo(clrThreadId: 0x7f4b48002a10, osThreadId: 298, osThreadHandle: 0x0xafc) completed for ProfilerThreadInfoId=42.
[2026-08-08 14:08:33.378 | debug | PId: 1 | TId: 298] Creating timer for thread 298
[2026-08-08 14:08:33.383 | debug | PId: 1 | TId: 181] Event pipe provider: System.Transactions.TransactionsEventSource
```
But this part is completely absent from the aks pod, and it's not sending any data at all.

Both of those have identical straces mentioning pyroscope:

```
openat(AT_FDCWD, "/app/Pyroscope.Linux.ApiWrapper.x64.so", O_RDONLY|O_CLOEXEC) = 3
[pid 380] openat(AT_FDCWD, "/app/Pyroscope.Profiler.Native.so", O_RDONLY|O_CLOEXEC) = 12
[pid 380] openat(AT_FDCWD, "/var/log/pyroscope/dotnet/Pyroscope-DotNet-Profiler-Native-dotnet-380.log", O_WRONLY|O_CREAT|O_APPEND, 0666) = 12
```
Here are the logs produced by both, they seem identical to me.
[aks-pod-logs.txt](https://github.com/user-attachments/files/30857169/aks-pod-logs.txt) - those end abruptly mid-sentence.

[local-container-logs.txt](https://github.com/user-attachments/files/30857170/local-container-logs.txt)

Those are my envs:
```
COPY --chown=1000:1000 --from=build /src/Pyroscope.Linux.ApiWrapper.x64.so .
COPY --chown=1000:1000 --from=build /src/Pyroscope.Profiler.Native.so .

ENV CORECLR_ENABLE_PROFILING=1
ENV CORECLR_PROFILER={BD1A650D-AC5D-4896-B64F-D6FA25D6B26A}
ENV CORECLR_PROFILER_PATH=/app/Pyroscope.Profiler.Native.so
ENV LD_PRELOAD=/app/Pyroscope.Linux.ApiWrapper.x64.so
ENV LD_LIBRARY_PATH=/app

ENV PYROSCOPE_APPLICATION_NAME=webuploads.dotnet.app
ENV PYROSCOPE_SERVER_ADDRESS=http://...
ENV PYROSCOPE_LOG_LEVEL=debug
ENV PYROSCOPE_PROFILING_ENABLED=1
ENV PYROSCOPE_PROFILING_ALLOCATION_ENABLED=true
ENV PYROSCOPE_PROFILING_CONTENTION_ENABLED=true
ENV PYROSCOPE_PROFILING_EXCEPTION_ENABLED=true
```

I have made sure that they're using the same architecture, and I've deleted any security context from the aks pod one.

Contributor guide

No contributing guide indexed for this repository

Research direction

Compare aks-pod-logs.txt with local-container-logs.txt, focusing on where the profiler output ends, then inspect the Dockerfile environment and the matching openat strace entries. Reproduce the profiler startup in an AKS pod and determine why the native profiler stops before producing profiles. Done means the cause is identified and profiling data is confirmed from the AKS pod.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, cpp, docker, kubernetes
Domain
devops, infrastructure, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.