nf-core / nf-core/modules

`manta/germline`: pinned container segfaults in `GenerateSVCandidates` on WGS trios

Open
#12,881 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Nextflow
Stars
429
Forks
1.1k
Avg merge
1d 6h
Merged PRs (30d)
153

Description

Description of the bug

The container pinned by manta/germline segfaults reproducibly part-way through
GenerateSVCandidates on a whole-genome trio. Illumina's official Manta 1.6.0
release build completes the identical workload. The difference is the build, not
the data.

Discovered through nf-core/raredisease 3.1.2, whose CALL_SV_MANTA subworkflow
runs this module. raredisease cannot skip Manta alone — skip_tools has no
manta entry and SVDB_MERGE requires one priority tag per VCF — so the only
available workaround is disabling the whole sv_calling subworkflow.

Container pinned
community.wave.seqera.io/library/manta_python:0eb71149179b3920

Steps to reproduce

Human WGS trio, GRCh38 (hg38.analysisSet.fa), ~30x per sample, aligned with
bwa-mem2 and duplicate-marked (stock nf-core/raredisease alignment; no ABRA2 or
external realignment). Joint three-sample invocation with the pinned
Clinical-Genomics call regions:

configManta.py --bam A.bam --bam B.bam --bam C.bam \
  --reference hg38.analysisSet.fa --runDir manta \
  --callRegions grch38_manta_call_regions_-1.0-.bed.gz
python manta/runWorkflow.py -m local -j 12

Observed

generateCandidateSV_0000 dies with SIGSEGV and no stderr:

[generateCandidateSV_0000] [wrapperSignal] taskExitCode -11
Last 0 stderr lines from task (of 0 total lines):

Four runs, three thread counts, same wall:

threads edges processed result
6 4806 SIGSEGV
12 4820 SIGSEGV
12 4819 SIGSEGV (enableRemoteReadRetrieval...=0)
1 4994 SIGSEGV (deterministic order, so not a race)

Expected

Completion, as with Illumina's official release build on identical inputs:

EXIT=0   368 tasks, 0 errors, 9h11m
results/variants/diploidSV.vcf.gz          18,789 records
results/variants/candidateSV.vcf.gz        7.1 MB
results/variants/candidateSmallIndels.vcf.gz  4.5 MB

Same BAMs, same reference, same call regions, same thread count, same host.

Ruled out by experiment

  • Memory / disk. Peak RSS 1.7 GB against a 36 GB request and 3.2 GB against
    72 GB. SLURM reports FAILED 1:0, not an OOM kill.
  • Thread race. Reproduces single-threaded at a comparable edge count.
  • enableRemoteReadRetrievalForInsertionsInGermlineCallingModes. Disabling
    it changes nothing.
  • chrM. chromDepth.txt reports chrM at 48697x against ~100x autosomal, and
    Manta's depthFilterMultiple guard is relative to each chromosome's own
    median so it never engages there — but a chrM-free call-regions BED crashed
    identically (4819 edges).
  • The locus graph. SummarizeSVLoci output is byte-identical between the
    two builds. Graph construction is correct in both; only candidate generation
    differs. (The two svLocusGraph.bin files differ in bytes and the official
    binary rejects the container's with FATAL_ERROR: unsupported version, but
    that is a Boost archive-format difference, not a semantic one.)
  • Input integrity. All BAMs pass samtools quickcheck; @PG chain is
    bwa-mem2 -> samtools -> MarkDuplicates -> samtools.

Suspected cause

Manta 1.6.0 vendors boost_1_58_0 in redist/ and is built and tested against
it. The two builds:

Pinned container Illumina official release
GenerateSVCandidates 10,310,880 bytes 7,621,512 bytes
Boost libboost_serialization.so.1.85.0, dynamic static, vendored 1.58
Compiler g++-13.3.0 g++-6.3.1
Build date 2025-02-04 2019-06-28

Manta has been unmaintained since 2019, so a 2025 rebuild of that C++ against a
Boost 27 minor versions newer is a plausible route from latent undefined
behaviour to a segfault. This was not isolated — compiler and htslib differ
too — but the remedy is the same regardless of which difference is responsible.

Suggested fix

quay.io/biocontainers/manta:1.6.0--h9ee0642_1 (2021-03-26) is the same build
shape as the official release: static Boost, GenerateSVCandidates 7,625,608
bytes against the official 7,621,512. Available as a Singularity image at
https://depot.galaxyproject.org/singularity/manta:1.6.0--h9ee0642_1.

The trade-off is an older htslib in that image. An alternative is a fresh
bioconda build pinned to Boost 1.58, which would need a bioconda-recipes
change.

Caveat for whoever triages this

This will most likely not reproduce with -profile test. The trigger looks
data-dependent: this trio's locus graph contains a pathological hub node at
chr2:32,915,715-32,917,079 (1,364 bp) with 39,924 edges against a
locus-wide median of 1 and p99 of 17, and 1,502,934 inbound observations —
nearly every node in an 84,061-node locus points at it. Subsampled test data is
very unlikely to build such a node. Reproducing this needs a real WGS trio.

Happy to supply the locus graph, SummarizeSVLoci output, or the full pyflow
logs on request.

Command used and terminal output

Through nf-core/raredisease.

samplesheet.csv:

sample,lane,fastq_1,fastq_2,sex,phenotype,paternal_id,maternal_id,case_id
NA10837,L001,ERR3989284_1.fastq.gz,ERR3989284_2.fastq.gz,1,2,NA12272,NA12273,G1K-trio
NA12272,L001,ERR3239465_1.fastq.gz,ERR3239465_2.fastq.gz,1,1,,,G1K-trio
NA12273,L001,ERR3239466_1.fastq.gz,ERR3239466_2.fastq.gz,2,1,,,G1K-trio

params.yaml (resource paths elided):

analysis_type: wgs
platform: illumina
genome: GRCh38
aligner: bwamem2
variant_caller: deepvariant
mito_name: chrM
skip_tools: gens,germlinecnvcaller
skip_subworkflows: snv_annotation,sv_annotation,mt_annotation,mt_subsample,me_calling,me_annotation,generate_clinical_set
nextflow run nf-core/raredisease -r 3.1.2 -profile singularity \
  --input samplesheet.csv --outdir results \
  -params-file params.yaml
Relevant files

1000 Genomes 30x on GRCh38 (NYGC), CEPH family 1418 trio. Full FASTQs, not
subsampled
— the on-disk sizes match ENA's published byte counts exactly.

Sample Role Run FASTQ
NA10837 proband (son) ERR3989284 _1.fastq.gz · _2.fastq.gz
NA12272 father ERR3239465 _1.fastq.gz · _2.fastq.gz
NA12273 mother ERR3239466 _1.fastq.gz · _2.fastq.gz

MD5s (ENA filereport API, result=read_run):

ERR3989284_1  5352a52cfa92cad5523435545088563c   12985369337 bytes
ERR3989284_2  8fe3b0655b7c1efd3ff2eb5e692aa1fa   14116751232 bytes
ERR3239465_1  cbf08408b0b10b6d491f6e235c8c7d80   13228284415 bytes
ERR3239465_2  c2a8eb74b6f34cc5cd8a2e0b67b201db   14233627994 bytes
ERR3239466_1  5c58f63696396dc7e18ea2c0f9a8e4a3   13195731876 bytes
ERR3239466_2  dc7b2757cd30951d2657d5d444c588b0   14284008497 bytes

Roughly 37x per sample, Illumina NovaSeq 6000, 2x150 bp; 82 GB of FASTQ total.
Data collection: https://www.internationalgenome.org/data-portal/data-collection/1000genomes_30x

Reference: UCSC hg38.analysisSet.fa (GCA_000001405.15).
Manta call regions: the pinned Clinical-Genomics artifact raredisease documents,
rare-disease/region/grch38_manta_call_regions_-1.0-.bed.gz at commit
eea53902baa56a05790fec49e256f22cf3454199
(sha256 fc5728b7f0e6fd18aa4a8015db4892cd9ba6cc7c97c6cf93589f2a313939cca5).

System information

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the manta/germline module and its pinned community.wave.seqera.io/library/manta_python:0eb71149179b3920 container, then inspect the GenerateSVCandidates entry point and the suggested Bioconda image. Reproduce with the listed full WGS trio and python manta/runWorkflow.py -m local -j 12; done means the workflow completes without SIGSEGV and the module uses a validated build.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
build-system, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.