grafana / grafana/pyroscope-dotnet
Pyroscope profiling doesn't work on aks pods
- Dominant language
- C++
- Stars
- 38
- Forks
- 13
- Avg merge
- 20h 15m
- Merged PRs (30d)
- 10
Description
Hi,
I am having a great problem with getting Pyroscope profiling to actually work in my aks pods.
I have successfully run the app image locally via `podman run` and it was working and sending the data to port-forwarded pyroscope in a proper way.
However, the same image, with the same exact envs is not producing profiles at all.
When starting the local container I see
```
[2026-08-08 14:08:33.366 | debug | PId: 1 | TId: 292] ManagedThreadList::SetThreadName(clrThreadId: 0x7f4b58010500, pThreadName: ".NET TP Worker") completed for ProfilerThreadInfoId=25.
[2026-08-08 14:08:33.368 | debug | PId: 1 | TId: 293] Callback invoked: ThreadCreated(threadId=0x7f4b58010500)
[2026-08-08 14:08:33.368 | debug | PId: 1 | TId: 293] Callback invoked: ThreadAssignedToOSThread(managedThreadId=0x7f4b58010500, osThreadId=293)
[2026-08-08 14:08:33.368 | debug | PId: 1 | TId: 293] ManagedThreadList::SetThreadOsInfo(clrThreadId: 0x7f4b58010500, osThreadId: 293, osThreadHandle: 0x0xab8) completed for ProfilerThreadInfoId=37.
[2026-08-08 14:08:33.368 | debug | PId: 1 | TId: 293] Creating timer for thread 293
[2026-08-08 14:08:33.368 | debug | PId: 1 | TId: 293] CorProfilerCallback::ThreadNameChanged(threadId=0x7f4b9818e180, name=".NET TP Worker")
[2026-08-08 14:08:33.368 | debug | PId: 1 | TId: 293] Map (0) to .NET TP Worker
[2026-08-08 14:08:33.368 | debug | PId: 1 | TId: 293] ManagedThreadList::SetThreadName(clrThreadId: 0x7f4b9818e180, pThreadName: ".NET TP Worker") completed for ProfilerThreadInfoId=26.
[2026-08-08 14:08:33.369 | debug | PId: 1 | TId: 294] Callback invoked: ThreadCreated(threadId=0x7f4b9818e180)
[2026-08-08 14:08:33.369 | debug | PId: 1 | TId: 294] Callback invoked: ThreadAssignedToOSThread(managedThreadId=0x7f4b9818e180, osThreadId=294)
[2026-08-08 14:08:33.369 | debug | PId: 1 | TId: 294] ManagedThreadList::SetThreadOsInfo(clrThreadId: 0x7f4b9818e180, osThreadId: 294, osThreadHandle: 0x0xacc) completed for ProfilerThreadInfoId=38.
[2026-08-08 14:08:33.369 | debug | PId: 1 | TId: 294] Creating timer for thread 294
[2026-08-08 14:08:33.369 | debug | PId: 1 | TId: 294] CorProfilerCallback::ThreadNameChanged(threadId=0x7f4b5c002a10, name=".NET TP Worker")
[2026-08-08 14:08:33.369 | debug | PId: 1 | TId: 294] Map (0) to .NET TP Worker
[2026-08-08 14:08:33.369 | debug | PId: 1 | TId: 294] ManagedThreadList::SetThreadName(clrThreadId: 0x7f4b5c002a10, pThreadName: ".NET TP Worker") completed for ProfilerThreadInfoId=27.
[2026-08-08 14:08:33.372 | debug | PId: 1 | TId: 295] Callback invoked: ThreadCreated(threadId=0x7f4b5c002a10)
[2026-08-08 14:08:33.372 | debug | PId: 1 | TId: 295] Callback invoked: ThreadAssignedToOSThread(managedThreadId=0x7f4b5c002a10, osThreadId=295)
[2026-08-08 14:08:33.372 | debug | PId: 1 | TId: 295] ManagedThreadList::SetThreadOsInfo(clrThreadId: 0x7f4b5c002a10, osThreadId: 295, osThreadHandle: 0x0xad8) completed for ProfilerThreadInfoId=39.
[2026-08-08 14:08:33.372 | debug | PId: 1 | TId: 295] Creating timer for thread 295
[2026-08-08 14:08:33.372 | debug | PId: 1 | TId: 295] CorProfilerCallback::ThreadNameChanged(threadId=0x7f4b50002a10, name=".NET TP Worker")
[2026-08-08 14:08:33.372 | debug | PId: 1 | TId: 295] Map (0) to .NET TP Worker
[2026-08-08 14:08:33.372 | debug | PId: 1 | TId: 295] ManagedThreadList::SetThreadName(clrThreadId: 0x7f4b50002a10, pThreadName: ".NET TP Worker") completed for ProfilerThreadInfoId=28.
[2026-08-08 14:08:33.373 | debug | PId: 1 | TId: 296] Callback invoked: ThreadCreated(threadId=0x7f4b50002a10)
[2026-08-08 14:08:33.373 | debug | PId: 1 | TId: 296] Callback invoked: ThreadAssignedToOSThread(managedThreadId=0x7f4b50002a10, osThreadId=296)
[2026-08-08 14:08:33.373 | debug | PId: 1 | TId: 296] ManagedThreadList::SetThreadOsInfo(clrThreadId: 0x7f4b50002a10, osThreadId: 296, osThreadHandle: 0x0xae4) completed for ProfilerThreadInfoId=40.
[2026-08-08 14:08:33.373 | debug | PId: 1 | TId: 296] Creating timer for thread 296
[2026-08-08 14:08:33.373 | debug | PId: 1 | TId: 296] CorProfilerCallback::ThreadNameChanged(threadId=0x7f4b54002a10, name=".NET TP Worker")
[2026-08-08 14:08:33.373 | debug | PId: 1 | TId: 296] Map (0) to .NET TP Worker
[2026-08-08 14:08:33.373 | debug | PId: 1 | TId: 296] ManagedThreadList::SetThreadName(clrThreadId: 0x7f4b54002a10, pThreadName: ".NET TP Worker") completed for ProfilerThreadInfoId=29.
[2026-08-08 14:08:33.374 | debug | PId: 1 | TId: 297] Callback invoked: ThreadCreated(threadId=0x7f4b54002a10)
[2026-08-08 14:08:33.374 | debug | PId: 1 | TId: 297] Callback invoked: ThreadAssignedToOSThread(managedThreadId=0x7f4b54002a10, osThreadId=297)
[2026-08-08 14:08:33.374 | debug | PId: 1 | TId: 297] ManagedThreadList::SetThreadOsInfo(clrThreadId: 0x7f4b54002a10, osThreadId: 297, osThreadHandle: 0x0xaf0) completed for ProfilerThreadInfoId=41.
[2026-08-08 14:08:33.374 | debug | PId: 1 | TId: 297] Creating timer for thread 297
[2026-08-08 14:08:33.374 | debug | PId: 1 | TId: 297] CorProfilerCallback::ThreadNameChanged(threadId=0x7f4b48002a10, name=".NET TP Worker")
[2026-08-08 14:08:33.374 | debug | PId: 1 | TId: 297] Map (0) to .NET TP Worker
[2026-08-08 14:08:33.374 | debug | PId: 1 | TId: 297] ManagedThreadList::SetThreadName(clrThreadId: 0x7f4b48002a10, pThreadName: ".NET TP Worker") completed for ProfilerThreadInfoId=2a.
[2026-08-08 14:08:33.378 | debug | PId: 1 | TId: 298] Callback invoked: ThreadCreated(threadId=0x7f4b48002a10)
[2026-08-08 14:08:33.378 | debug | PId: 1 | TId: 298] Callback invoked: ThreadAssignedToOSThread(managedThreadId=0x7f4b48002a10, osThreadId=298)
[2026-08-08 14:08:33.378 | debug | PId: 1 | TId: 298] ManagedThreadList::SetThreadOsInfo(clrThreadId: 0x7f4b48002a10, osThreadId: 298, osThreadHandle: 0x0xafc) completed for ProfilerThreadInfoId=42.
[2026-08-08 14:08:33.378 | debug | PId: 1 | TId: 298] Creating timer for thread 298
[2026-08-08 14:08:33.383 | debug | PId: 1 | TId: 181] Event pipe provider: System.Transactions.TransactionsEventSource
```
But this part is completely absent from the aks pod, and it's not sending any data at all.
Both of those have identical straces mentioning pyroscope:
```
openat(AT_FDCWD, "/app/Pyroscope.Linux.ApiWrapper.x64.so", O_RDONLY|O_CLOEXEC) = 3
[pid 380] openat(AT_FDCWD, "/app/Pyroscope.Profiler.Native.so", O_RDONLY|O_CLOEXEC) = 12
[pid 380] openat(AT_FDCWD, "/var/log/pyroscope/dotnet/Pyroscope-DotNet-Profiler-Native-dotnet-380.log", O_WRONLY|O_CREAT|O_APPEND, 0666) = 12
```
Here are the logs produced by both, they seem identical to me.
[aks-pod-logs.txt](https://github.com/user-attachments/files/30857169/aks-pod-logs.txt) - those end abruptly mid-sentence.
[local-container-logs.txt](https://github.com/user-attachments/files/30857170/local-container-logs.txt)
Those are my envs:
```
COPY --chown=1000:1000 --from=build /src/Pyroscope.Linux.ApiWrapper.x64.so .
COPY --chown=1000:1000 --from=build /src/Pyroscope.Profiler.Native.so .
ENV CORECLR_ENABLE_PROFILING=1
ENV CORECLR_PROFILER={BD1A650D-AC5D-4896-B64F-D6FA25D6B26A}
ENV CORECLR_PROFILER_PATH=/app/Pyroscope.Profiler.Native.so
ENV LD_PRELOAD=/app/Pyroscope.Linux.ApiWrapper.x64.so
ENV LD_LIBRARY_PATH=/app
ENV PYROSCOPE_APPLICATION_NAME=webuploads.dotnet.app
ENV PYROSCOPE_SERVER_ADDRESS=http://...
ENV PYROSCOPE_LOG_LEVEL=debug
ENV PYROSCOPE_PROFILING_ENABLED=1
ENV PYROSCOPE_PROFILING_ALLOCATION_ENABLED=true
ENV PYROSCOPE_PROFILING_CONTENTION_ENABLED=true
ENV PYROSCOPE_PROFILING_EXCEPTION_ENABLED=true
```
I have made sure that they're using the same architecture, and I've deleted any security context from the aks pod one.
Contributor guide
No contributing guide indexed for this repository
Research direction
Compare aks-pod-logs.txt with local-container-logs.txt, focusing on where the profiler output ends, then inspect the Dockerfile environment and the matching openat strace entries. Reproduce the profiler startup in an AKS pod and determine why the native profiler stops before producing profiles. Done means the cause is identified and profiling data is confirmed from the AKS pod.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, cpp, docker, kubernetes
- Domain
- devops, infrastructure, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100