nf-core / nf-core/variantbenchmarking

unexpected F1 score running somatic snv/indel variants benchmarking

Open
#225 10 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Nextflow
Stars
51
Forks
31
Avg merge
1d 22h
Merged PRs (30d)
1

Description

Description of the bug

Hi,

I am running the following
nextflow run nf-core/variantbenchmarking
-r 1.3.0
-c $inDIR/yale.config
-work-dir $workDIR
--input $inDIR/${smplshtFILE}.csv
--outdir $outPATH/$outDIR
--analysis $analysisTYPE
--variant_type snv
--method sompy \
--truth_id ${smpID}
--truth_vcf $refPATH/${refFILE}.vcf.gz
--preprocess split_multiallelic,normalize,deduplicate,prepy
--fasta $GENOME
--fai $GENOMEIDX

I am testing a VCF derived from RNAseq and one from WGS, against the same WGS. Again, this is to have a sort of control

The resulting sompy.summary.csv is below
Tool,File,Type,TP_base,TP,FN,TP_call,FP,UNK,Recall,Precision,recall_lower,recall_upper,recall2,precision_lower,precision_upper,na,ambiguous,fp.region.size,F1
RNAsentieonGATK,RNAsentieonGATK.661T.GATK.stats.csv,SNVs,5,1,4,67,66,0,0.2,0.0149253731343283,0.0,0.6286264006319943,0.2,0.0,0.0676321414491473,0.0,0.0,675359842.0,0.0977256802900045
WGSsentieonGATK,WGSsentieonGATK.661T.GATK.stats.csv,SNVs,5,5,0,5,0,0,1.0,1.0,0.4781762498950185,1.0,1.0,0.4781762498950185,1.0,0.0,0.0,675359842.0,0.0

The second row VCF is the same as the reference. Beyond the
recall_lower,recall_upper,recall2,precision_lower,precision_upper which I will have to read about, I have F1=0, but Recall and Precision are both 1.0

Am I doing something. wrong ?

Thanks

Command used and terminal output

Relevant files

No response

System information

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the supplied Nextflow command and inspect the generated sompy.summary.csv, starting with how F1 is calculated from TP, FP, FN, Recall, and Precision. No relevant files or tests are identified; done means the discrepancy is explained and the benchmark reports a consistent F1 value or documents why it is 0.

Written by the indexing model from the issue text.

Assessment

Domain
bioinformatics
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.