scverse / scverse/rustar-aligner

GzDecoder truncates multi-member gzip input: reads are silently dropped

Open
#217 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Rust
Stars
75
Forks
7
Avg merge
8m
Merged PRs (30d)
1

Description

Summary

Gzipped input is decoded with flate2::read::GzDecoder, which stops at the end of the first
gzip member
. A multi-member .gz is therefore read partially, silently: no error, no warning,
just fewer reads. flate2::read::MultiGzDecoder is the variant that consumes concatenated members.

This matters in practice because multi-member gzip is normal in this domain: bcl2fastq output,
any cat a.fq.gz b.fq.gz > merged.fq.gz, and every BGZF file (BGZF is multi-member gzip).

Reproduction

On main, using only the fixtures bundled in the repo:

# 3 reads total, split across two gzip members
head -8 test/reads.fq | gzip -c  > /tmp/multi.fq.gz
tail -4 test/reads.fq | gzip -c >> /tmp/multi.fq.gz
# same 3 reads, one member
gzip -c test/reads.fq > /tmp/single.fq.gz

cargo run -- --runMode genomeGenerate --genomeDir /tmp/idx \
    --genomeFastaFiles test/simple.fa --genomeSAindexNbases 4
for f in single multi; do
  cargo run -- --genomeDir /tmp/idx --readFilesIn /tmp/$f.fq.gz \
      --outFileNamePrefix /tmp/out_$f/ --outSAMtype BAM SortedByCoordinate
  grep 'Number of input reads' /tmp/out_$f/Log.final.out
done
Input gunzip -c | wc -l Number of input reads
single.fq.gz 12 3
multi.fq.gz 12 2

Both files decompress to the same 12 lines with gunzip/Python. The aligner sees one read fewer,
and exits 0.

Affected sites

All four read paths use the single-member decoder:

  • src/io/fastq.rs:106 — FASTQ input (the one that matters most)
  • src/solo/whitelist.rs:524 — barcode whitelist
  • src/solo/count.rs:2090
  • src/bin/emptydrops.rs:92

Fix

GzDecoderMultiGzDecoder at those four sites. No API change, no new dependency; the type is
already in the flate2 version in the tree.

Suggested regression test

An integration test that builds a two-member .fq.gz from the bundled fixture and asserts the read
count matches the plain-text run. Cheap, and it locks the behaviour against a future refactor
picking the wrong decoder again.

Note

Detection is also extension-based today (path_str.ends_with(".gz"), src/io/fastq.rs:92), so a
compressed file with an unexpected name is read as plain text and a plain file named .gz fails.
That is a separate question, tracked with the multi-format input issue.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the four affected sites: src/io/fastq.rs:106, src/solo/whitelist.rs:524, src/solo/count.rs:2090, and src/bin/emptydrops.rs:92. Run the supplied two-member FASTQ reproduction and compare its read count with the plain-text or single-member result. Done means all four paths consume concatenated gzip members and a regression test confirms the complete read count.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
bioinformatics
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.