nextflow-io / nextflow-io/nextflow

Allow a list of keys in the `by` option of the v2 version of the `join` operator

Open
#7,309 4 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

lang/dataflow triage/investigate
Dominant language
Groovy
Stars
3.5k
Forks
811
Avg merge
2d 11h
Merged PRs (30d)
61

Description

New feature

Currently the by option in the .join operator for typed workflows allows only a single key to be specified. This on it's own is a very useful feature and will greatly improve how we write pipelines. It would however also be helpful if this option could take a list of multiple keys and only join entries where all of these keys match. This would prevent pipeline developers from having to do a .map on both channels before joining them. It would also increase the readability of the pipeline as it's clear from the join operator what keys are used exactly (instead of one custom key created just for joining)

Use case

Joining channels with records that have multiple keys in common

Suggested implementation

ch1.join(ch2, by['id', 'type']) or similar

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the v2 typed-workflow .join operator and trace how its current single-key by option is parsed and matched. Review the existing join tests for one key, then add coverage showing that a list such as ['id', 'type'] joins entries only when every key matches; done means the multi-key form works without requiring preceding .map operations.

Written by the indexing model from the issue text.

Assessment

Tech stack
groovy
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.