scverse / scverse/rustar-aligner
GzDecoder truncates multi-member gzip input: reads are silently dropped
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 75
- Forks
- 7
- Avg merge
- 8m
- Merged PRs (30d)
- 1
Description
Summary
Gzipped input is decoded with flate2::read::GzDecoder, which stops at the end of the first
gzip member. A multi-member .gz is therefore read partially, silently: no error, no warning,
just fewer reads. flate2::read::MultiGzDecoder is the variant that consumes concatenated members.
This matters in practice because multi-member gzip is normal in this domain: bcl2fastq output,
any cat a.fq.gz b.fq.gz > merged.fq.gz, and every BGZF file (BGZF is multi-member gzip).
Reproduction
On main, using only the fixtures bundled in the repo:
# 3 reads total, split across two gzip members
head -8 test/reads.fq | gzip -c > /tmp/multi.fq.gz
tail -4 test/reads.fq | gzip -c >> /tmp/multi.fq.gz
# same 3 reads, one member
gzip -c test/reads.fq > /tmp/single.fq.gz
cargo run -- --runMode genomeGenerate --genomeDir /tmp/idx \
--genomeFastaFiles test/simple.fa --genomeSAindexNbases 4
for f in single multi; do
cargo run -- --genomeDir /tmp/idx --readFilesIn /tmp/$f.fq.gz \
--outFileNamePrefix /tmp/out_$f/ --outSAMtype BAM SortedByCoordinate
grep 'Number of input reads' /tmp/out_$f/Log.final.out
done
| Input | gunzip -c | wc -l |
Number of input reads |
|---|---|---|
single.fq.gz |
12 | 3 |
multi.fq.gz |
12 | 2 |
Both files decompress to the same 12 lines with gunzip/Python. The aligner sees one read fewer,
and exits 0.
Affected sites
All four read paths use the single-member decoder:
src/io/fastq.rs:106— FASTQ input (the one that matters most)src/solo/whitelist.rs:524— barcode whitelistsrc/solo/count.rs:2090src/bin/emptydrops.rs:92
Fix
GzDecoder → MultiGzDecoder at those four sites. No API change, no new dependency; the type is
already in the flate2 version in the tree.
Suggested regression test
An integration test that builds a two-member .fq.gz from the bundled fixture and asserts the read
count matches the plain-text run. Cheap, and it locks the behaviour against a future refactor
picking the wrong decoder again.
Note
Detection is also extension-based today (path_str.ends_with(".gz"), src/io/fastq.rs:92), so a
compressed file with an unexpected name is read as plain text and a plain file named .gz fails.
That is a separate question, tracked with the multi-format input issue.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the four affected sites: src/io/fastq.rs:106, src/solo/whitelist.rs:524, src/solo/count.rs:2090, and src/bin/emptydrops.rs:92. Run the supplied two-member FASTQ reproduction and compare its read count with the plain-text or single-member result. Done means all four paths consume concatenated gzip members and a regression test confirms the complete read count.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- bioinformatics
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100