`manta/germline`: pinned container segfaults in `GenerateSVCandidates` on WGS trios
Nobody has claimed this yet.
- Dominant language
- Nextflow
- Stars
- 429
- Forks
- 1.1k
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 153
Description
Description of the bug
The container pinned by manta/germline segfaults reproducibly part-way through
GenerateSVCandidates on a whole-genome trio. Illumina's official Manta 1.6.0
release build completes the identical workload. The difference is the build, not
the data.
Discovered through nf-core/raredisease 3.1.2, whose CALL_SV_MANTA subworkflow
runs this module. raredisease cannot skip Manta alone — skip_tools has no
manta entry and SVDB_MERGE requires one priority tag per VCF — so the only
available workaround is disabling the whole sv_calling subworkflow.
Container pinned
community.wave.seqera.io/library/manta_python:0eb71149179b3920
Steps to reproduce
Human WGS trio, GRCh38 (hg38.analysisSet.fa), ~30x per sample, aligned with
bwa-mem2 and duplicate-marked (stock nf-core/raredisease alignment; no ABRA2 or
external realignment). Joint three-sample invocation with the pinned
Clinical-Genomics call regions:
configManta.py --bam A.bam --bam B.bam --bam C.bam \
--reference hg38.analysisSet.fa --runDir manta \
--callRegions grch38_manta_call_regions_-1.0-.bed.gz
python manta/runWorkflow.py -m local -j 12
Observed
generateCandidateSV_0000 dies with SIGSEGV and no stderr:
[generateCandidateSV_0000] [wrapperSignal] taskExitCode -11
Last 0 stderr lines from task (of 0 total lines):
Four runs, three thread counts, same wall:
| threads | edges processed | result |
|---|---|---|
| 6 | 4806 | SIGSEGV |
| 12 | 4820 | SIGSEGV |
| 12 | 4819 | SIGSEGV (enableRemoteReadRetrieval...=0) |
| 1 | 4994 | SIGSEGV (deterministic order, so not a race) |
Expected
Completion, as with Illumina's official release build on identical inputs:
EXIT=0 368 tasks, 0 errors, 9h11m
results/variants/diploidSV.vcf.gz 18,789 records
results/variants/candidateSV.vcf.gz 7.1 MB
results/variants/candidateSmallIndels.vcf.gz 4.5 MB
Same BAMs, same reference, same call regions, same thread count, same host.
Ruled out by experiment
- Memory / disk. Peak RSS 1.7 GB against a 36 GB request and 3.2 GB against
72 GB. SLURM reportsFAILED 1:0, not an OOM kill. - Thread race. Reproduces single-threaded at a comparable edge count.
enableRemoteReadRetrievalForInsertionsInGermlineCallingModes. Disabling
it changes nothing.- chrM.
chromDepth.txtreports chrM at 48697x against ~100x autosomal, and
Manta'sdepthFilterMultipleguard is relative to each chromosome's own
median so it never engages there — but a chrM-free call-regions BED crashed
identically (4819 edges). - The locus graph.
SummarizeSVLocioutput is byte-identical between the
two builds. Graph construction is correct in both; only candidate generation
differs. (The twosvLocusGraph.binfiles differ in bytes and the official
binary rejects the container's withFATAL_ERROR: unsupported version, but
that is a Boost archive-format difference, not a semantic one.) - Input integrity. All BAMs pass
samtools quickcheck;@PGchain is
bwa-mem2 -> samtools -> MarkDuplicates -> samtools.
Suspected cause
Manta 1.6.0 vendors boost_1_58_0 in redist/ and is built and tested against
it. The two builds:
| Pinned container | Illumina official release | |
|---|---|---|
GenerateSVCandidates |
10,310,880 bytes | 7,621,512 bytes |
| Boost | libboost_serialization.so.1.85.0, dynamic |
static, vendored 1.58 |
| Compiler | g++-13.3.0 | g++-6.3.1 |
| Build date | 2025-02-04 | 2019-06-28 |
Manta has been unmaintained since 2019, so a 2025 rebuild of that C++ against a
Boost 27 minor versions newer is a plausible route from latent undefined
behaviour to a segfault. This was not isolated — compiler and htslib differ
too — but the remedy is the same regardless of which difference is responsible.
Suggested fix
quay.io/biocontainers/manta:1.6.0--h9ee0642_1 (2021-03-26) is the same build
shape as the official release: static Boost, GenerateSVCandidates 7,625,608
bytes against the official 7,621,512. Available as a Singularity image at
https://depot.galaxyproject.org/singularity/manta:1.6.0--h9ee0642_1.
The trade-off is an older htslib in that image. An alternative is a fresh
bioconda build pinned to Boost 1.58, which would need a bioconda-recipes
change.
Caveat for whoever triages this
This will most likely not reproduce with -profile test. The trigger looks
data-dependent: this trio's locus graph contains a pathological hub node at
chr2:32,915,715-32,917,079 (1,364 bp) with 39,924 edges against a
locus-wide median of 1 and p99 of 17, and 1,502,934 inbound observations —
nearly every node in an 84,061-node locus points at it. Subsampled test data is
very unlikely to build such a node. Reproducing this needs a real WGS trio.
Happy to supply the locus graph, SummarizeSVLoci output, or the full pyflow
logs on request.
Command used and terminal output
Through nf-core/raredisease.
samplesheet.csv:
sample,lane,fastq_1,fastq_2,sex,phenotype,paternal_id,maternal_id,case_id
NA10837,L001,ERR3989284_1.fastq.gz,ERR3989284_2.fastq.gz,1,2,NA12272,NA12273,G1K-trio
NA12272,L001,ERR3239465_1.fastq.gz,ERR3239465_2.fastq.gz,1,1,,,G1K-trio
NA12273,L001,ERR3239466_1.fastq.gz,ERR3239466_2.fastq.gz,2,1,,,G1K-trio
params.yaml (resource paths elided):
analysis_type: wgs
platform: illumina
genome: GRCh38
aligner: bwamem2
variant_caller: deepvariant
mito_name: chrM
skip_tools: gens,germlinecnvcaller
skip_subworkflows: snv_annotation,sv_annotation,mt_annotation,mt_subsample,me_calling,me_annotation,generate_clinical_set
nextflow run nf-core/raredisease -r 3.1.2 -profile singularity \
--input samplesheet.csv --outdir results \
-params-file params.yaml
Relevant files
1000 Genomes 30x on GRCh38 (NYGC), CEPH family 1418 trio. Full FASTQs, not
subsampled — the on-disk sizes match ENA's published byte counts exactly.
| Sample | Role | Run | FASTQ |
|---|---|---|---|
| NA10837 | proband (son) | ERR3989284 | _1.fastq.gz · _2.fastq.gz |
| NA12272 | father | ERR3239465 | _1.fastq.gz · _2.fastq.gz |
| NA12273 | mother | ERR3239466 | _1.fastq.gz · _2.fastq.gz |
MD5s (ENA filereport API, result=read_run):
ERR3989284_1 5352a52cfa92cad5523435545088563c 12985369337 bytes
ERR3989284_2 8fe3b0655b7c1efd3ff2eb5e692aa1fa 14116751232 bytes
ERR3239465_1 cbf08408b0b10b6d491f6e235c8c7d80 13228284415 bytes
ERR3239465_2 c2a8eb74b6f34cc5cd8a2e0b67b201db 14233627994 bytes
ERR3239466_1 5c58f63696396dc7e18ea2c0f9a8e4a3 13195731876 bytes
ERR3239466_2 dc7b2757cd30951d2657d5d444c588b0 14284008497 bytes
Roughly 37x per sample, Illumina NovaSeq 6000, 2x150 bp; 82 GB of FASTQ total.
Data collection: https://www.internationalgenome.org/data-portal/data-collection/1000genomes_30x
Reference: UCSC hg38.analysisSet.fa (GCA_000001405.15).
Manta call regions: the pinned Clinical-Genomics artifact raredisease documents,
rare-disease/region/grch38_manta_call_regions_-1.0-.bed.gz at commit
eea53902baa56a05790fec49e256f22cf3454199
(sha256 fc5728b7f0e6fd18aa4a8015db4892cd9ba6cc7c97c6cf93589f2a313939cca5).
System information
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the manta/germline module and its pinned community.wave.seqera.io/library/manta_python:0eb71149179b3920 container, then inspect the GenerateSVCandidates entry point and the suggested Bioconda image. Reproduce with the listed full WGS trio and python manta/runWorkflow.py -m local -j 12; done means the workflow completes without SIGSEGV and the module uses a validated build.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- build-system, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100