max_unpack BCF_UN_STR SIGSEGV, other strange behavior
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- c
- Domain
- bioinformatics
Research direction
Start with the minimal reproducer and the max_unpack documentation in vcf.h. Trace bcf_dup, bcf_unpack, bcf_translate, and bcf_write into vcf.c, especially vcf_format at line 2739, while comparing b->d.n_flt with bnew->d.n_flt. Done means the reproducer no longer produces incorrect FILTER values or a SIGSEGV when using BCF_UN_STR.
Written by the indexing model from the issue text.
Description
Hi, thanks for htslib and how responsive developers have always been. Apologies in advance for the long text below.
While exploring potential speedups setting bcf1_t.max_unpack as documented in vcf.h:
int max_unpack; // Set to BCF_UN_STR, BCF_UN_FLT, or BCF_UN_INFO to boost performance of vcf_parse when some of the fields won't be needed
I found that by setting to BCF_UN_STR (in contrast to all other values, which seem to work as expected), I can:
A. get wrong behavior in some fields (in particular, multiple FILTER results)
B. generate occasional to guaranteed segfaults ( with invalid bcf1_t->d.n_flt value )
Since BCF_UN_STR by definition does not include the FILTER field, and segfaults are related to n_flt/FILTER, I do suspect this is bug, but perhaps I am making calls in wrong order, or using max_unpack incorrectly -- please let me know! Below is a minimal reproducible example.
Notes:
- setting MAX_UNPACK to any other value seems to operate as expected
- by adding additional header lines to output, you can increase the probability of segfault.
- disabling
b->max_unpack = <whatever>prior to callingbcf_readprevents segfault - disabling
bcf_unpack(bnew, MAX_UNPACK)inside the loop increases probability of segfault to 100%: will sigsegv at vcf.c:vcf_format:2739 when trying to kputs a filter that DNE (looking at stack, n_flt is always some larger number when it should be 0 or 1) - whether or not
bcf_unpackis called, the FILTER column exhibits strange artifacts (multiple when there should not be, mostly) - removing
bnewand reusing onlybeliminates issue, see comment at bottom
// copy one vcf file to another
#include <stdio.h>
#include <htslib/vcf.h>
#define MAX_UNPACK BCF_UN_STR
int main(void)
{
vcfFile *infile = bcf_open("resources/gnomad.chrY.vcf", "r");
vcfFile *outfile = bcf_open("/tmp/out.vcf", "w");
bcf_hdr_t* infile_hdr = bcf_hdr_read(infile);
bcf_hdr_t* outfile_hdr = bcf_hdr_dup(infile_hdr);
// uncomment below to increase frequency of segfault; likewise you can add a second header line to increase probabilty further
bcf_hdr_append(outfile_hdr, "##contig=<ID=chrZ,length=57227415,source=whatever>");
bcf_hdr_write(outfile, outfile_hdr);
bcf1_t *b = bcf_init1();
b->max_unpack = MAX_UNPACK;
while( bcf_read(infile, infile_hdr, b) >= 0 )
{
bcf1_t *bnew = bcf_dup(b);
int ret = bcf_unpack(bnew, MAX_UNPACK);
bcf_translate(outfile_hdr, infile_hdr, bnew);
if (bnew->d.n_flt > 1) break;
bcf_write(outfile, outfile_hdr, bnew); // SIGSEGV in this call tree
bcf_destroy(bnew);
bcf_empty(b);
}
bcf_close(infile);
bcf_close(outfile);
return 0;
}
By removing bnew and only operating on b (without even calling bcf_empty) the problem disappears entirely and program works as expected. bnew is included because this C code recapitulates exactly the calls made by an htslib wrapper in a higher level OOP language, for which I do need to copy the bcf1_t.
Thanks again, and if I am calling the API in wrong way or out of order , please, let me know!
- Dominant language
- C
- Stars
- 950
- Forks
- 475
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 11
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from samtools/htslib
-
Difficulty 3/5 1-2 days Newbie friendliness 74/100
-
Difficulty 3/5 1-2 days Newbie friendliness 76/100
-
tabix performance Open
Similar issues
-
[adam] AdamNet network read doesn't cap to MAX_ADAM_PACKET_LEN, overflows client receive buffers Open
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
FujiNetWIFI/fujinet-firmware#1649 · 2 comments ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
HarbourMasters/Shipwright#7229 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
riscv-software-src/riscv-isa-sim#2435 · 1 comment ·
-
bug Self Built Image SNAPSHOT Supported Device target/ramips
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100