open-telemetry / open-telemetry/opentelemetry-cpp

TlsRandomNumberGenerator causes memory corruption when the process is forked.

Open
#2,408 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug do-not-stale triage/accepted
Dominant language
C++
Stars
1.4k
Forks
632
Avg merge
1d 13h
Merged PRs (30d)
75

Description

I manage an open source project https://github.com/hpcc-systems/HPCC-Platform which we adding support to open telemetry using this library.

One of the components launches large numbers (100+) of child processes at the same time. Since adding the open telemetry library this is now the system to core when those child processes are started. Here is an example of a stack trace from gdb:

#0  0x00007fc40090eb5d in read () from /usr/lib64/libc.so.6
#1  0x00007fc40089ad54 in __GI__IO_file_underflow () from /usr/lib64/libc.so.6
#2  0x00007fc400899808 in __GI__IO_file_xsgetn () from /usr/lib64/libc.so.6
#3  0x00007fc40088e1ef in fread () from /usr/lib64/libc.so.6
#4  0x00007fc4011baa50 in std::random_device::_M_getval() () from /usr/lib64/libstdc++.so.6
#5  0x00007fc40183995b in opentelemetry::v1::sdk::common::(anonymous namespace)::TlsRandomNumberGenerator::Seed() ()
   from /opt/HPCCSystems/lib/libopentelemetry_common.so
#6  0x00007fc4008e4c4e in fork () from /usr/lib64/libc.so.6
#7  0x00007fc4036aeb69 in CLinuxPipeProcess::run (this=0x7fc352cb63a0)
    at /hpcc-dev/HPCC-Platform/system/jlib/jthread.cpp:2072

I believe the problem is caused by the following code:

  TlsRandomNumberGenerator() noexcept
  {
    Seed();
    platform::AtFork(nullptr, nullptr, OnFork);
  }

where an onFork handler was added inside the random number generator made in 2018.

There are only a very restricted set of functions that are valid to be called at that point within the child process (https://man7.org/linux/man-pages/man3/pthread_atfork.3.html): they must be async-signal-safe, and not use any heap functions because that can cause memory corruption. This onFork call does not obey those restrictions. It also performs unnecessary work whenever you create a child process (I suspect it now takes a long time to start the child processes because of the need to wait for sufficient entropy from the random number generator).

I believe the fix is to delete that call to onFork, and re-examine why that change was made. In particular, why would a process ever call the open telemetry functions inside the child of a fork()?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at TlsRandomNumberGenerator's constructor and its platform::AtFork/OnFork handling, using the provided fork stack trace and pthread_atfork guidance as context. Reproduce the failure with the fork-heavy HPCC-Platform scenario, then verify that child-process startup no longer triggers memory corruption or unnecessary random-number reseeding.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.