grafana / grafana/pyroscope-python

Fork safety

Aperta
#122 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Rust
Stelle
4
Fork
2
Merge medio
2g 4h
PR unite (30g)
5

Descrizione

During review of https://github.com/grafana/pyroscope-python/pull/83 we found our profiler is not safe for use if the target application is using forks after the pyroscope python SDK has started few threads.

A good description of the class of the problem can be found here https://www.qnx.com/developers/docs/8.0/com.qnx.doc.neutrino.getting_started/topic/s1_procs_Multithreaded_fork.html

> POSIX says that following a [fork()](https://www.qnx.com/developers/docs/8.0/com.qnx.doc.neutrino.lib_ref/topic/f/fork.html), the child's memory state is a copy of the parent's, but the child contains only a single thread; any other threads in the parent aren't duplicated in the child. If one of those parent threads was in the middle of manipulating a data structure when the fork() happened, that data structure will be in an undefined, possibly inconsistent, state in the child following the fork(). Regular mutexes can be used to protect different threads from interfering with each other, but protecting against fork() requires the forking code to be aware of the internals of your data structure synchronization.
>
> For example, suppose the parent process has two threads, T1 and T2, and T2 has locked a mutex. If T1 calls fork(), the child process is a copy of the parent, including any mutexes, but with only one thread. If the thread in the child tries to lock the mutex, the operation fails because T2 has it locked—and there is no T2 in the child process, so no one can unlock the mutex. This can lead to a deadlock.
>
> The simplest solution is to call fork() only from a single-threaded process, or to call it before you create additional threads. But, if you insist on using fork() in a multithreaded process, there are some things you can do:
>
> - Use at-fork handlers
> - Use forksafe mutexes
> - Call exec*() soon after forking. POSIX requires that the child process use only functions that are async-signal-safe until it calls one of the exec*() functions.

Another good read https://discuss.python.org/t/switching-default-multiprocessing-context-to-spawn-on-posix-as-well/21868

We've attempted to https://github.com/grafana/pyroscope-python/pull/119 . Few problems were found and workarounded, such as logging stderr lock, glibc dns resolver (only fixed on linux, not mac), http client internal locking(probably from tokio). The running agent memory was leaked in the child. There is at least one mutex that was not addressed in this PR inside the openssl errors table (see the PR description).

The cpython does attempt to recover the interpreter state in the child, but issues a warning about deadlocks. https://github.com/python/cpython/issues/100228

It has been decided to postpone the attempts to recover forked children and abandon #119 until we see need. Instead we're going to issue a warning and document this as unsupported and invalid use of the pyroscope python sdk with an examples how to avoid these problems (start the pyroscope after the forks, or stop the pyroscope before forking and ensure no threads are interacting with pyroscope during the fork

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Direzione di ricerca

Start by reviewing pull request #119 and the SDK's fork-related behavior described in this issue. Identify where the profiler starts and stops its threads and where user-facing warnings and documentation are maintained. Done means warning users that forked-child use is unsupported and documenting the safe alternatives: fork before starting the profiler, or stop it before forking.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
python, rust
Ambito
documentation, observability-sre
Tipo di issue
Documentazione
Difficoltà
3/5
Tempo stimato
1-2 giorni
Stato di attività
Tranquilla
Chiarezza
Abbastanza chiara
Idoneità per principianti
55/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.