grafana / grafana/pyroscope-python

Investigate musllinux-arm64 CI test hangs for Python 3.9, 3.10, 3.11

Open
#36 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
4
Forks
2
Avg merge
2d 4h
Merged PRs (30d)
5

Description

Description

In the linux-test CI matrix added in grafana/pyroscope-rs#364, the musllinux-arm64 variant hangs indefinitely for Python 3.9, 3.10, and 3.11, while 3.12 and 3.13 complete successfully (~37s each).

All four variants started at the same time. The three older versions never complete.

Observed behaviour

Job Status Duration
musllinux-arm64 - 3.9 stuck in_progress 15+ min
musllinux-arm64 - 3.10 stuck in_progress 15+ min
musllinux-arm64 - 3.11 stuck in_progress 15+ min
musllinux-arm64 - 3.12 completed ~37s
musllinux-arm64 - 3.13 completed ~37s

Example run: https://github.com/grafana/pyroscope-rs/actions/runs/22293868489

Possible causes

  • python:3.9-alpine, python:3.10-alpine, python:3.11-alpine images may not have an arm64 variant on Docker Hub, causing docker pull to hang or fail silently
  • py-spy may not support profiling on Alpine arm64 for older Python versions
  • The test watchdog (120s timeout) fires but the process doesn't exit cleanly inside the container

Current workaround

These three combinations are excluded from the matrix via exclude: in ci-ffi-python.yml until the root cause is identified.

Next steps

  • Check Docker Hub for python:3.9-alpine / 3.10-alpine / 3.11-alpine arm64 manifest availability
  • Run the docker pull manually on an arm64 runner to confirm
  • Check if py-spy works on Alpine arm64 for these Python versions

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with ci-ffi-python.yml and the linked Actions run, then check Docker Hub manifests for the Python 3.9–3.11 Alpine arm64 images. Reproduce the pulls and py-spy test on an arm64 runner, checking the 120-second watchdog behavior. Done means the hang’s cause is identified and the exclusion or CI configuration can be updated accordingly.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, github-actions, python
Domain
ci-cd, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.