nextflow-io / nextflow-io/nextflow

Feature request: pubish per-channel entry index files

Open
#6,830 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Groovy
Stars
3.5k
Forks
811
Avg merge
2d 11h
Merged PRs (30d)
61

Description

New feature

Currently, when publishing an index with the new output block, the output index is an index across all the channel entries published. However, especially with the new Record types coming, it would be nice to be able to publish a per-channel index, as the record/map being published may have its own complicated metadata structure that is worth being recorded at a sample level.

Use case

Say you are publishing a record/map that agglomerates a bunch of information about a sample, and you want to publish the sample files to a specific sample directory in the output. One might also want to publish a sample-level index file in that captures sample-level metadata and files together in that directory.

Suggested implementation

Maybe this would need an extension to add an entryIndex or similar directive to a publishing block:

output {
    sample {
        path { s ->
            s.fastq >> "${s.id}/fastq/"
            s.bam >> "${s.id}/bam/{s.sorted ? "sorted" : "not_sorted"}.bam"
        }
        entryIndex { s ->
            path "${s.id}/index.json"
        }
    }
}

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the existing output-block index publishing path and its tests; the issue does not name files or entry points. Define and validate per-channel entryIndex behavior for sample metadata and files, including the proposed path syntax, with tests covering the published index contents.

Written by the indexing model from the issue text.

Assessment

Tech stack
groovy
Domain
tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.