bcftools-1.2.1 filter --targets-file outputs duplicated entries

Reproduce the command using bcftools 1.2, the provided BED regions, the compressed VCF, and its CSI index. Start with filter --targets-file and the target-region/index handling, then compare streamed output with direct decompression. Done means identifying the cause of duplicated records and documenting or correcting the stale-index warning behavior.

This may be an issue with my BED file containing some duplicate regions as I used

$ bcftools filter --targets-file S04380110__Agilent_SureSelect_Human_All_Exon_V5/hg38/S04380110_Covered.bed -O v mysample.samtools.filtered.vcf.bgz | grep 85543981
Warning: The index file is older than the data file: mysample.samtools.filtered.vcf.bgz.csi
chr15   85543981        .       C       T       228     PASS    DP=28;VDB=0.467145;SGB=-0.692067;MQSB=1;MQ0F=0;AC=2;AN=2;DP4=0,0,6,14;MQ=60     GT:PL   1/1:255,60,0
[W::vcf_parse] contig '.' is not defined in the header. (Quick workaround: index the file with tabix.)
chr15   85543981        .       C       T       228     PASS    DP=28;VDB=0.467145;SGB=-0.692067;MQSB=1;MQ0F=0;AC=2;AN=2;DP4=0,0,6,14;MQ=60     GT:PL   1/1:255,60,0
^C
$
$ gzip -dc  mysample.samtools.filtered.vcf.bgz | grep 85543981
chr15   85543981        .       C       T       228     .       DP=28;VDB=0.467145;SGB=-0.692067;MQSB=1;MQ0F=0;AC=2;AN=2;DP4=0,0,6,14;MQ=60     GT:PL   1/1:255,60,0

gzip: mysample.samtools.filtered.vcf.bgz: decompression OK, trailing garbage ignored
$
$ grep 85543981 mysample.samtools.filtered.vcf
chr15   85543981        .       C       T       228     .       DP=28;VDB=0.467145;SGB=-0.692067;MQSB=1;MQ0F=0;AC=2;AN=2;DP4=0,0,6,14;MQ=60     GT:PL   1/1:255,60,0
$

The message Warning: The index file is older than the data file: mysample.samtools.filtered.vcf.bgz.csi should IMHO speak about either mysample.samtools.filtered.vcf.bgz or even mysample.samtools.filtered.vcf.

Anyway, I conclude the input files (.vcf.bgz nor .vcf) for 'filter --targets-file' did not contain duplicates. I cannot inspect the CSI index for duplicates.

It could be the issue is triggered by the CSI index warning message somehow, or the index contains duplicated info somehow, or the BED file contains some overlapping or even same=duplicate regions which trigger the duplicates in output of the streaming 'filter --targets-file' mode.

I do not see the CSI index having an older timestamp than the BGZ file:

-rw-r----- 1 mmokrejs mmokrejs  10107830 Dec 22 17:58 mysample.samtools.filtered.vcf
-rw-r----- 1 mmokrejs mmokrejs    236254 Dec 22 17:58 mysample.samtools.filtered.vcf.bgz.csi
-rw-r----- 1 mmokrejs mmokrejs 817960421 Dec 22 17:58 mysample.samtools.filtered.vcf.bgz

Are here are 3 lines from the BED file around the example duplicate entry:

chr15   85533636        85533915        ref|AKAP13,ref|NM_007200,ref|NM_006738,ens|ENST00000361243,ens|ENST00000560302,ens|ENST00000559362,ens|ENST00000394518,mRNA|AK125331,mRNA|AF387101,mRNA|KJ902219,mRNA|JA482237,mRNA|AB05589
0,mRNA|AF406992,mRNA|BC172356,mRNA|BC050312,mRNA|JA482238,mRNA|CU690639,mRNA|CU690638,ccds|CCDS32319.1,ccds|CCDS32320.1
chr15   85543764        85544158        ref|AKAP13,ref|NM_007200,ref|NM_006738,ens|ENST00000361243,ens|ENST00000558166,ens|ENST00000559362,ens|ENST00000394518,mRNA|AK125331,mRNA|AF387101,mRNA|JA482237,mRNA|AB055890,mRNA|AF40699
2,mRNA|BC172356,mRNA|JA482238,ccds|CCDS32319.1,ccds|CCDS32320.1
chr15   85555337        85555526        ref|AKAP13,ref|NM_007200,ref|NM_006738,ens|ENST00000361243,ens|ENST00000558166,ens|ENST00000559362,ens|ENST00000394518,mRNA|AK125331,mRNA|AF387101,mRNA|JA482237,mRNA|AB055890,mRNA|AF40699
2,mRNA|BC172356,mRNA|JA482238,ccds|CCDS32319.1,ccds|CCDS32320.1

If you have some smarter way how to extract relevant lines from the BED file please let me know.

$ bcftools --version
bcftools 1.2
Using htslib 1.2.1