nf-core / nf-core/website

[FEATURE] A standing list of file extensions/file types that should be compressed

Open
#3,181 6 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

documentation
Dominant language
MDX
Stars
98
Forks
298
Avg merge
14h 38m
Merged PRs (30d)
38

Description

Is your feature request related to a problem? Please describe

Somewhat related to a problem, but also at the same time not really. I totally agree with compressed files should be used.

Where applicable, the usage and generation of compressed files SHOULD be enforced as input and output, respectively:

*.fastq.gz and NOT *.fastq
*.bam and NOT *.sam

I'm just not sure where to draw the line. The big ones (FASTQ and BAM) are straight forward, but I think it becomes more difficult for things like BLAST results, genome annotations, log files, etc...

I started a conversation in Slack, but think putting it here might be better.

Describe the solution you'd like

I'm wondering if there could be a standing list of extensions that should be compressed. Here's an example using Prokka outputs:

.err   failed annotations
.faa   proteins fasta
.ffn   genes fasta
.fna   contigs
.fsa   contigs
.gbk   genbank file
.gff   gff3 annotations
.log   prokka outputs
.sqn   sequin file
.tbl   tbl2asn file
.tsv   tsv of annotations
.txt   annotation stats 

I think most of these should be compressed, especially the FASTA ones. This is where I think a standing list of extensions, or maybe file types (e.g. sequences in FASTA format), would be useful as a guide and make the choice easier on submitters.

Here's a working list (big overlap with Prokka):

.aln   alignment
.fa    fasta
.faa   proteins fasta
.fasta fasta
.fastq fastq
.fq    fastq
.ffn   genes fasta
.fna   contigs
.fsa   contigs
.gbk   genbank file
.gfa   assembly graph
.gff   gff3 annotations
.sqn   sequin file
.tbl   tbl2asn file
.vcf   variants

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names no files, tests, or entry points. Start by reviewing the proposed extension lists and the related Slack discussion, then determine where a standing compression guide belongs and what file types it should cover. Done means an agreed list or file-type policy is documented for submitters.

Written by the indexing model from the issue text.

Assessment

Domain
documentation
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.