eclipse-iceoryx / eclipse-iceoryx/iceoryx

Semaphore overflow

Open
#2,477 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
2.2k
Forks
492
Avg merge
18h 57m
Merged PRs (30d)
1

Description

**Operating system:**
Debian12

@elBoberido Good afternoon, I ran into this problem, when working for more than 2 hours for a long time, I get the following error

```
июл 20 16:56:22 yadro ngworker[2993412]: 2025-07-20 16:56:22.890 [Error]: /builds/ngfw/thirdparty/iceoryx/iceoryx_hoofs/posix/sync/source/semaphore_interface.cpp:56 { iox::expected iox::detail::SemaphoreInterface::post() [with SemaphoreChild = iox::UnnamedSemaphore] -> iox_sem_post } ::: [ 75 ] Value too large for defined data type
июл 20 16:56:22 yadro ngworker[2993412]: 2025-07-20 16:56:22.890 [Error]: Semaphore overflow. The maximum value of 2147483647 would be exceeded.
июл 20 16:56:22 yadro ngworker[2993412]: 2025-07-20 16:56:22.890 [Fatal]: /builds/ngfw/thirdparty/iceoryx/iceoryx_posh/source/popo/building_blocks/condition_notifier.cpp:44 [Fatal Error] [POPO__CONDITION_NOTIFIER_SEMAPHORE_CORRUPT_IN_NOTIFY (code = 78)] in module [iceoryx_posh (id = 2)]
июл 20 16:56:22 yadro audit[2993412]: ANOM_ABEND auid=4294967295 uid=1002 gid=1003 ses=4294967295 subj=0:0:0:0:0 pid=2993412 comm="engine" exe="/usr/bin/ngworker" sig=6 res=1
июл 20 16:56:22 yadro audit[2993466]: ANOM_ABEND auid=4294967295 uid=1002 gid=1003 ses=4294967295 subj=0:0:0:0:0 pid=2993466 comm="engine" exe="/usr/bin/ngworker" sig=6 res=1
июл 20 16:56:22 yadro audit[2993654]: ANOM_ABEND auid=4294967295 uid=1002 gid=1003 ses=4294967295 subj=0:0:0:0:0 pid=2993654 comm="engine" exe="/usr/bin/ngworker" sig=6 res=1
июл 20 16:56:22 yadro ngworker[2993654]: 2025-07-20 16:56:22.892 [Error]: /builds/ngfw/thirdparty/iceoryx/iceoryx_hoofs/posix/sync/source/mutex.cpp:330 { iox::expected iox::mutex::lock() -> iox_pthread_mutex_lock } ::: [ 130 ] Owner died
июл 20 16:56:22 yadro ngworker[2993654]: 2025-07-20 16:56:22.892 [Error]: The thread/process which owned the mutex died. The mutex is now in an inconsistent state and must be put into a consistent state again with Mutex::make_consistent()
июл 20 16:56:22 yadro ngworker[2993654]: 2025-07-20 16:56:22.892 [Fatal]: Locking of an inter-process mutex failed! This indicates that the application holding the lock was terminated or the resources were cleaned up by RouDi due to an unresponsive application.
июл 20 16:56:22 yadro ngworker[2993654]: 2025-07-20 16:56:22.892 [Fatal]: /builds/ngfw/thirdparty/iceoryx/iceoryx_posh/source/popo/building_blocks/locking_policy.cpp:42 [Fatal Error] [POPO__CHUNK_LOCKING_ERROR (code = 61)] in module [iceoryx_posh (id = 2)]
июл 20 16:56:22 yadro ngworker[2993654]: 2025-07-20 16:56:22.892 [Fatal]: /builds/ngfw/thirdparty/iceoryx/iceoryx_posh/source/popo/building_blocks/locking_policy.cpp:42 [PANIC]
июл 20 16:56:22 yadro audit[2993340]: ANOM_ABEND auid=4294967295 uid=1002 gid=1003 ses=4294967295 subj=0:0:0:0:0 pid=2993340 comm="ccproc.3" exe="/usr/bin/context-core" sig=6 res=1
июл 20 16:56:22 yadro audit[2995005]: ANOM_ABEND auid=4294967295 uid=1002 gid=1003 ses=4294967295 subj=0:0:0:0:0 pid=2995005 comm="engine" exe="/usr/bin/ngworker" sig=6 res=1
июл 20 16:56:22 yadro ngworker[2993412]: 2025-07-20 16:56:22.890 [Fatal]: /builds/ngfw/thirdparty/iceoryx/iceoryx_posh/source/popo/building_blocks/condition_notifier.cpp:44 [PANIC]
июл 20 16:56:22 yadro ngworker[2993466]: 2025-07-20 16:56:22.891 [Error]: /builds/ngfw/thirdparty/iceoryx/iceoryx_hoofs/posix/sync/source/mutex.cpp:330 { iox::expected iox::mutex::lock() -> iox_pthread_mutex_lock } ::: [ 130 ] Owner died
июл 20 16:56:22 yadro ngworker[2993466]: 2025-07-20 16:56:22.891 [Error]: The thread/process which owned the mutex died. The mutex is now in an inconsistent state and must be put into a consistent state again with Mutex::make_consistent()
июл 20 16:56:22 yadro ngworker[2993466]: 2025-07-20 16:56:22.891 [Fatal]: Locking of an inter-process mutex failed! This indicates that the application holding the lock was terminated or the resources were cleaned up by RouDi due to an unresponsive application.
июл 20 16:56:22 yadro ngworker[2993466]: 2025-07-20 16:56:22.891 [Fatal]: /builds/ngfw/thirdparty/iceoryx/iceoryx_posh/source/popo/building_blocks/locking_policy.cpp:42 [Fatal Error] [POPO__CHUNK_LOCKING_ERROR (code = 61)] in module [iceoryx_posh (id = 2)]
июл 20 16:56:22 yadro ngworker[2993466]: 2025-07-20 16:56:22.891 [Fatal]: /builds/ngfw/thirdparty/iceoryx/iceoryx_posh/source/popo/building_blocks/locking_policy.cpp:42 [PANIC]
июл 20 16:56:22 yadro audit[2994885]: ANOM_ABEND auid=4294967295 uid=1002 gid=1003 ses=4294967295 subj=0:0:0:0:0 pid=2994885 comm="engine" exe="/usr/bin/ngworker" sig=6 res=1
июл 20 16:56:22 yadro ngworker[2995005]: 2025-07-20 16:56:22.893 [Error]: /builds/ngfw/thirdparty/iceoryx/iceoryx_hoofs/posix/sync/source/mutex.cpp:330 { iox::expected iox
::mutex::lock() -> iox_pthread_mutex_lock } ::: [ 130 ] Owner died
июл 20 16:56:22 yadro ngworker[2995005]: 2025-07-20 16:56:22.893 [Error]: The thread/process which owned the mutex died. The mutex is now in an inconsistent state and must be put into a consistent s
tate again with Mutex::make_consistent()
июл 20 16:56:22 yadro ngworker[2995005]: 2025-07-20 16:56:22.893 [Fatal]: Locking of an inter-process mutex failed! This indicates that the application holding the lock was terminated or the resourc
es were cleaned up by RouDi due to an unresponsive application.
июл 20 16:56:22 yadro ngworker[2995005]: 2025-07-20 16:56:22.893 [Fatal]: /builds/ngfw/thirdparty/iceoryx/iceoryx_posh/source/popo/building_blocks/locking_policy.cpp:42 [Fatal Error] [POPO__CHUNK_LO
CKING_ERROR (code = 61)] in module [iceoryx_posh (id = 2)]
июл 20 16:56:22 yadro ngworker[2995005]: 2025-07-20 16:56:22.893 [Fatal]: /builds/ngfw/thirdparty/iceoryx/iceoryx_posh/source/popo/building_blocks/locking_policy.cpp:42 [PANIC]
июл 20 16:56:22 yadro audit[2994223]: ANOM_ABEND auid=4294967295 uid=1002 gid=1003 ses=4294967295 subj=0:0:0:0:0 pid=2994223 comm="engine" exe="/usr/bin/ngworker" sig=6 res=1
июл 20 16:56:22 yadro context-core[2993340]: 2025-07-20 16:56:22.893 [Error]: /builds/ngfw/thirdparty/iceoryx/iceoryx_hoofs/posix/sync/source/mutex.cpp:330 { iox::expected
iox::mutex::lock() -> iox_pthread_mutex_lock } ::: [ 130 ] Owner died
июл 20 16:56:22 yadro context-core[2993340]: 2025-07-20 16:56:22.893 [Error]: The thread/process which owned the mutex died. The mutex is now in an inconsistent state and must be put into a consiste
nt state again with Mutex::make_consistent()
июл 20 16:56:22 yadro context-core[2993340]: 2025-07-20 16:56:22.893 [Fatal]: Locking of an inter-process mutex failed! This indicates that the application holding the lock was terminated or the res
ources were cleaned up by RouDi due to an unresponsive application.
```

I'm sending the event like this:
```
static thread_local auto current_tid = gettid();
auto try_res = publishers.find(current_tid);
if (try_res == publishers.end()) {
try {
auto [fst, snd] = publishers.emplace(std::piecewise_construct, std::forward_as_tuple(current_tid),
std::forward_as_tuple(iox::capro::ServiceDescription{
"SES", {TruncateToCapacity_t(), context.empty() ? "_all_" : context.c_str()}, "Event"}));
try_res = fst;
} catch (std::exception const& ex) {
sesc_logger.StdException(::boost::core::demangle(typeid(ex).name()), ex.what());
return StatusPublish::ERROR;
} catch (...) {
return StatusPublish::ERROR;
}
}

try_res->second.publish(event_buffer);
```

I get the event like this:
```
SesConverter::SesConverter(::logs::app& logger, std::string const& module, std::shared_ptr statistics,
std::shared_ptr processor)
: m_iox_runtime{iox::runtime::PoshRuntime::initRuntime(
{TruncateToCapacity_t{},
module == "_all_" ? sextfunc::Cef::APP_NAME : ::std::string(sextfunc::Cef::APP_NAME).append("_").append(module).c_str()})},
m_statistics{std::move(statistics)},
m_processor{std::move(processor)},
m_logger{logger},
m_iox_listener{::std::make_unique<::iox::popo::Listener>()},
m_ses_subscriber{new iox::popo::UntypedSubscriber(iox::capro::ServiceDescription{"SES", {TruncateToCapacity_t(), module.c_str()}, "Event"},
iox::popo::SubscriberOptions{.queueCapacity = 100000, .historyRequest = 10})} {
m_iox_listener->attachEvent(*m_ses_subscriber, iox::popo::SubscriberEvent::DATA_RECEIVED, iox::popo::createNotificationCallback(ses_event, *this))
.or_else([](auto) {
std::cerr << "unable to attach subscriber_ses" << std::endl;
std::exit(EXIT_FAILURE);
});
}

... other code

while (subscriber->take().and_then([subscriber, ses_converter](auto& sample) {
::ses::des::DeserBuffer deser_buffer(sample);
[[maybe_unused]] auto const event_id = deser_buffer.get();
switch_event(event_id, *ses_converter, deser_buffer);
subscriber->release(sample);
})) {
}
```

I really want to understand what needs to be done so that there is no problem, I need your help :))

Contributor guide

Open the contributing guide

Research direction

Start by reading iceoryx_hoofs/posix/sync/source/semaphore_interface.cpp at the failing post() path, then trace the notification flow through iceoryx_posh/source/popo/building_blocks/condition_notifier.cpp. Compare that with the publisher and subscriber lifecycles shown in the report, including queueCapacity and sample release. Done means reproducing the long-running overflow on Debian 12 and establishing a validated change that prevents the fatal failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, debian
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.