nextflow-io / nextflow-io/nextflow

Add toTopic() channel operator for workflow-level topic assignment

Open
#6,756 5 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

lang/dataflow stale
Dominant language
Groovy
Stars
3.5k
Forks
811
Avg merge
2d 11h
Merged PRs (30d)
61

Description

Problem

Topics currently can only be written to via process output: blocks with topic: name. There's no way to forward an existing channel to a topic from a workflow context.

This forces a choice between:

  1. Modifying upstream modules to add topic: outputs (breaks separation of concerns)
  2. Creating relay processes that exist solely to forward data to topics (adds complexity/overhead)
  3. Maintaining explicit emit chains through subworkflow hierarchies (extreme verbosity)
Use Case: nf-core/rnaseq

We're migrating nf-core/rnaseq to use workflow outputs (Nextflow 25.10+). The pipeline has ~130 outputs flowing through 3-4 levels of subworkflows. Each output requires:

Process emit → Subworkflow emit → Subworkflow emit → Workflow emit → Publish

This creates thousands of lines of channel wiring. We've even hit JVM string constant limits, forcing architectural changes purely to reduce code volume.

nf-core modules correctly emit outputs without knowledge of publishing—they shouldn't need to know about topics. But calling workflows should be able to route those outputs to topics trivially.

Proposed Solution

A channel operator to forward channel contents to a topic:

// Forward a channel to a topic
STAR_ALIGN.out.bam.toTopic('alignment_bams')
STAR_ALIGN.out.log_final.toTopic('star_logs')

// Later, in entry workflow publish block
publish:
alignment_bams = channel.topic('alignment_bams')
star_logs = channel.topic('star_logs')
Benefits
  1. Separation of concerns: Modules emit normally, workflows control routing
  2. Eliminates emit chains: No more passing outputs through every subworkflow layer
  3. Natural aggregation: Multiple sources can forward to the same topic
  4. Backwards compatible: Existing topic: in process outputs continues to work
Prior Discussion

This was requested in Discussion #2842 (2022) but never implemented.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the existing process output topic behavior, channel operators, workflow-level output wiring, and the publish block described in the issue. Define how a channel-level toTopic() would interact with channel.topic(), multiple sources, subworkflows, and existing topic outputs; done means the behavior is specified, implemented, and covered by relevant tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
groovy
Domain
tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.