eclipse-iceoryx / eclipse-iceoryx/iceoryx

Race condition between ProbeProcessAliveWithSigTerm in RouDi and SignalWatcher singleton in applications

Open
#2,308 5 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
C++
Stars
2.2k
Forks
492
Avg merge
18h 57m
Merged PRs (30d)
1

Description

## Required information
When shutting down applications, RouDi periodically sends SIGTERM signals. Usually these are all eaten by the SignalWatcher in applications which simply sets a flag that a SIGTERM was received.

But there is a very small window of time where right after the destructor for the SignalWatcher has run and reset the signal handler, but before the process is actually dead, that the SIGTERM triggers the default signal handler which terminates the program with a non-zero exit code.

This is pretty annoying when monitoring applications for non-zero exit codes because every once in awhile you'll get the SIGTERM exit code (usually 128 + 15 = 143 though in my environment it seems to be just 15).

**Operating system:**
NVidia DriveOS 6.0.6

**Compiler version:**
Can't recall off the top of my head - some version of gcc for arm64 CPU

**Eclipse iceoryx version:**
Fork based off of 8a5e08348a8fe830cdb1d92bf5b299a3e4bc8282 (Upstream mean as of 2024.11.1)

**Observed result or behaviour:**
Occasionally applications have exit code 15, but all signs from RouDi and the application are that it exited gracefully

**Expected result or behaviour:**
Exit code 0 always. Preferably we monitor for processes being alive or not some other way besides spamming SIGTERM. I believe the idea of a file lock was mentioned w.r.t. Iceoryx2?

**Conditions where it occurred / Performed steps:**
Have an application which connects to RouDi and uses the `waitForTermination()` signal handler method. Take a non-trivial amount of time to shutdown such that RouDi sends several followup SIGTERM's after the first. Be unlucky and have the signal watcher partially destroyed when receiving one of those SIGTERMS

## Additional helpful information

Contributor guide

Open the contributing guide

Research direction

Start by tracing ProbeProcessAliveWithSigTerm in RouDi and the SignalWatcher singleton used by applications, especially the waitForTermination() path. Reproduce shutdown with a non-trivial teardown and repeated SIGTERM signals, then verify that applications consistently exit with code 0 without the signal handler being triggered after SignalWatcher destruction.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.