regression of mpileup performance on Nanopore

Start with the reported v1.12 versus v1.13 mpileup comparison on Nanopore data, using the Illumina and PacBio CCS truth data and pairwise SNP-distance measurements described. Reproduce the recall and precision regression while varying the mentioned BAQ, Nanopore configuration, and genotype-likelihood parameters; done means identifying a reproducible cause or actionable configuration.

I have seen some differences in recall and precision with the big changes in the mpileup command in v1.13 with Nanopore data.

The first thing to say though is v1.13 is a massive improvement in terms of both CPU and memory usage. 🔥

I'm not sure of the best place to start this discussion so I'll start with a vague description of the scenario and I can provide more details for what you feel is relevant.

I am performing some SNP calling precision and recall analysis for Mycobacterium tuberculosis samples which I have Illumina and Nanopore data for. Seven of these sample I also have PacBio CCS for so I am used the CCS assemblies as truth genomes and masking spurious locations in further analysis.

I had fine-tuned some filters for version <=1.12 that provided precision on par with Illumina and recall a few percentage points lower than Illumina. However, these filters fail miserably for v1.13 - recall drops off a cliff, but precision stays good.

In addition to the seven samples with CCS assemblies, I have another 143 samples with Illumina and Nanopore data. My proxy for comparing these is pairwise SNP distance between samples for both technologies. The distance between a pair of samples should be the same on both technologies, so any difference are technology-driven. With v1.13 I am seeing a huge drop in the pairwise distance between samples on Nanopore - i.e., recall is very bad.

I have tried playing around with almost all of the mpileup parameters I think might be useful - full BAQ, partial BAQ, I tried the nanopore config, and basically all of the genotype likelihood parameters. I basically can't find a set of filters that provide a good balance between precision and recall for v1.13.

Normally I would just ignore this and stick with v1.12. But I figured this is likely helpful for others, and v1.12 uses an insane amount of memory and takes a few hours, compared to v1.13 which uses nearly no memory and CPU time comparatively.

Apologies for the rambling nature of this issue. I'm happy to provide plots, some data, try out some parameters etc.