open-rmf / open-rmf/crossflow

Make async service wind-down more robust

Open
#180 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
72
Forks
16
PR merge metrics
No merged PRs in 30d

Description

The execution of async services is among the most complicated parts of this library. One issue which recently surfaced is that there is a small risk of a race condition in the reachability detection when a workflow relies on a stream coming out of an async service in order to reach the terminate operation. The race condition is this:

  1. If the async service sends a message out over its stream
  2. That streamed message is placed into the execution channel as an unresolved command
  3. The async service finishes and notifies its scope that it should test reachability
  4. Reachability is tested while the streamed message is still an unresolved command within the execution channel
  5. The reachability check fails because we don't realize that some unresolved command is going to eventually provide a new message to the scope

Currently we avoid this problem by only testing reachability after we've made sure the entire execution channel is drained of all commands. Theoretically this should work fine, but it's fragile. We may accidentally tweak the implementation of the flushing system without regard for this, or some other similar race condition may appear in the future that we haven't accounted for.

We should consider strategies to change how async services themselves are executed so that other parts of the code don't have to be sensitive to them. For example, we could track lifecycle of commands in the execution channel, and we only allow an async service to conclude after all its commands have been flushed from the channel and the channel itself is gone. This could also address https://github.com/open-rmf/crossflow/issues/29 because we would be sure that the service is not seen as "finished" until the channel has dropped.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing async service execution, the execution channel, command flushing, and reachability detection. Reproduce the race where a streamed message remains unresolved during the reachability check, then evaluate command lifecycle and channel closure behavior. Done means service completion is not observed until its commands are flushed and the channel is gone, while also addressing the related issue referenced as #29.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.