Guard BitVectorHelper.getNullCount in StructVector.nullMaskOrNull
Open
@zaleslaw is already working on this.
Since Sep 8, 2026.
files
performance
- Dominant language
- Kotlin
- Stars
- 1.1k
- Forks
- 83
- Avg merge
- 4d 12h
- Merged PRs (30d)
- 30
Description
Noticed that Total runtime of reading a bunch of parquet datasets goes from 55 sec in rc01 to 70 sec in dev.
Main difference comes from BitVectorHelper.getNullCount usage in new code. It actually goes over entire vector to count, even though no nulls are expected in StructVector according to schema
Extra check on StructVector nullability seems to achieve the same result, assuming we can trust file schema to not mislead us and not have sneaky nulls:
val nullMask = if (field.isNullable) vector.nullMaskOrNull(range) else null
then nullCount > 0 should not be needed then and reading time goes back to 55s.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.