nextflow-io / nextflow-io/nextflow

Should `notifyEvent` be dispatched asynchronously?

Open
#7,189 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Groovy
Stars
3.5k
Forks
811
Avg merge
2d 11h
Merged PRs (30d)
61

Description

In #6471, I proposed making notifyEvent asynchronous, as originally suggested by @pditommaso in #6443:

Likely those events should be emitted async in this context to avoid that core runtime performance could be impacted by compute/data intensive operations that could be implemented by third-party plugins.

The PR was closed by mistake, I think, because the reason provided is not correct. However. the underlying question was left open: should notifyEvent dispatches be asynchronous?

I'm raising this again because users of the nf-lamin plugin are encountering slow pipeline execution times that appear to be caused by synchronous API calls made inside observer callbacks (e.g. onFilePublish). Since these calls happen on the main Nextflow thread, they block pipeline progress. Making notifyEvent async would eliminate this class of problem for all third-party plugins without requiring workarounds in each one.

I'm happy to reopen or rebase #6471 if there's interest. Alternatively, if the preference is for plugins to manage their own threading, we can do that, but it would just be good to know the intended design so we can align accordingly.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the context in #6471 and #6443, then trace the notifyEvent dispatch path and observer callbacks such as onFilePublish. Determine whether asynchronous dispatch is the intended design or whether plugins should manage threading themselves. Done means the project has a decided direction and the behavior is documented and covered appropriately.

Written by the indexing model from the issue text.

Assessment

Tech stack
groovy
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.