nextflow-io / nextflow-io/nextflow
Add toTopic() channel operator for workflow-level topic assignment
Nobody has claimed this yet.
- Dominant language
- Groovy
- Stars
- 3.5k
- Forks
- 811
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 61
Description
Problem
Topics currently can only be written to via process output: blocks with topic: name. There's no way to forward an existing channel to a topic from a workflow context.
This forces a choice between:
- Modifying upstream modules to add
topic:outputs (breaks separation of concerns) - Creating relay processes that exist solely to forward data to topics (adds complexity/overhead)
- Maintaining explicit emit chains through subworkflow hierarchies (extreme verbosity)
Use Case: nf-core/rnaseq
We're migrating nf-core/rnaseq to use workflow outputs (Nextflow 25.10+). The pipeline has ~130 outputs flowing through 3-4 levels of subworkflows. Each output requires:
Process emit → Subworkflow emit → Subworkflow emit → Workflow emit → Publish
This creates thousands of lines of channel wiring. We've even hit JVM string constant limits, forcing architectural changes purely to reduce code volume.
nf-core modules correctly emit outputs without knowledge of publishing—they shouldn't need to know about topics. But calling workflows should be able to route those outputs to topics trivially.
Proposed Solution
A channel operator to forward channel contents to a topic:
// Forward a channel to a topic
STAR_ALIGN.out.bam.toTopic('alignment_bams')
STAR_ALIGN.out.log_final.toTopic('star_logs')
// Later, in entry workflow publish block
publish:
alignment_bams = channel.topic('alignment_bams')
star_logs = channel.topic('star_logs')
Benefits
- Separation of concerns: Modules emit normally, workflows control routing
- Eliminates emit chains: No more passing outputs through every subworkflow layer
- Natural aggregation: Multiple sources can forward to the same topic
- Backwards compatible: Existing
topic:in process outputs continues to work
Prior Discussion
This was requested in Discussion #2842 (2022) but never implemented.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the existing process output topic behavior, channel operators, workflow-level output wiring, and the publish block described in the issue. Define how a channel-level toTopic() would interact with channel.topic(), multiple sources, subworkflows, and existing topic outputs; done means the behavior is specified, implemented, and covered by relevant tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- groovy
- Domain
- tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100