eclipse-iceoryx / eclipse-iceoryx/iceoryx

Ability to wake up WaitSets using a file descriptor instead of a condition variable

Open
#1,045 8 comments 0 reactions 0 assignees View on GitHub
feature request question
Dominant language
C++
Stars
2.2k
Forks
492
Avg merge
18h 57m
Merged PRs (30d)
1

Description

## Brief feature description

Iceoryx' WaitSets use condition variables to be notified. Unfortunately standard UNIX event loops do not support waiting on condition variables (condvars) along with other file descriptors. This makes it difficult to use Iceoryx in an event loop like libev/libuv.

## ToDos

1. [ ] Introduce additional template parameter so that a custom condition variable type can be used for the waitset
2. [ ] Implement a posix abstraction for the file descriptor set
3. [ ] Implement a condition variable based on file descriptors
4. [ ] Integrate file descriptor based condition variable into iceoryx so that waitset and listener are no longer based on semaphores
5. [ ] Platform performance improvements: Map the `FD_SET`/`select` API on linux to the `epoll` API and on QNX to the `ionotify` API.

## Detailed information

It is quite common on UNIX systems to use an event loop based around the select()/poll()/epoll() system calls, monitoring a set of file descriptors. This can be used to process I/O on the descriptors. In robotics, such a loop could be reading from a camera for example. It is possible to notify this event loop about general outstanding work using a pipe() or eventfd() (Linux-only) coupled with a producer-consumer queue. Other threads can add work to the queue, then write to the pipe/eventfd. This will wake up the event loop and it can read the queue for work.

It would be nice to build on these primitives to process Iceoryx messages in the same event loop. POSIX condition variables are not file descriptors and cannot be plugged into these system calls. However, they can be placed in shared memory, such that other processes (not just threads) can signal the condvar. WaitSets are implemented using condvars. This means, to use Iceoryx in a system with event loops, we have to start a separate thread, run the WaitSet there, and then use the producer-consumer queue to relay messages from Iceoryx to the event loop. However, this introduces extra context switches, system calls and data copying, which is undesirable.

Unfortunately file descriptors (fds) are not identical between processes and cannot be stored in shared memory. It is however possible for a process to send file descriptors to another process over a domain socket. The fd may have another number in the child process, but it will refer to the same process.

Considering that RouDi already uses a domain socket to coordinate with processes using Iceoryx, it should be possible for RouDi to create file descriptors on behalf of each interested thread, and coordinate to send them to other processes. Let me elaborate:

Consider Roudi, and ProcessA and ProcessB.
ProcessA has a publisher for a channel, and ProcessB has 2 threads, one of which (Thread2) has a subscriber for that channel.

What we would like is for ProcessA to be able to wake up Thread2 in a file descriptor event loop world.

To accomplish this, we need to do a little dance.

Either RouDi or ProcessB need to create a file descriptor. They would then need to exchange the fd over the domain socket so that both are aware of the fd that will be able to wake up (a specific thread in) ProcessB.

Now, ProcessA will have to receive that file descriptor from RouDi, and maintain a mapping in its process that it should write to this fd after subscriptions have been updated that ProcessB is interested in.

All of this is rather complicated, and perhaps doesn't mesh with the Iceoryx architecture, where the shared memory bits are fairly independent of individual processes, but I wanted to kick start the discussion.

I'm not sure how Iceoryx processes currently know which condition variables need to be notified when events arrive. Perhaps it doesn't go through condvars at all, and the WaitSet is responsible for noticing changes to a subscription within the process boundary and then updating its condvar. In that case, it might be easier to fit with file descriptors.

Interested in knowing if this is viable and how one would go about it. If it seems reasonable, I could consider writing the code.

Contributor guide

Open the contributing guide

Research direction

The issue names no files or tests. Start by tracing WaitSet and listener condition-variable notification, then review RouDi's domain-socket coordination; the proposed work spans custom condition variables, POSIX file-descriptor abstraction, integration, and platform-specific performance, so done requires an agreed architecture and completion of the relevant ToDos.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, linux
Domain
distributed-systems, operating-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.