cucapra / cucapra/pollen

Pangenotype matrix: use a dense bit vector

Open
#245 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
45
Forks
3
PR merge metrics
No merged PRs in 30d

Description

Currently (as of #238), constructing a pangenotype matrix produces a `Vec>`. We could make this more memory-efficient by using a big bit vector. Fortunately, we already depend on [the bit-vec crate](https://docs.rs/bit-vec/latest/bit_vec/), so let's just use that.

This will require exposing a different interface to Python. We can consider using NumPy's [packbits](https://numpy.org/doc/stable/reference/generated/numpy.packbits.html) and its complement to translate back and forth between the dense format and usable matrices.

As part of this effort, we need to figure out how to benchmark the pangenotype matrix construction; I would like to know how much faster this optimization makes things go.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the pangenotype matrix construction introduced in #238 and trace its Rust representation through the Python interface. Review the existing bit-vec dependency and investigate NumPy packbits conversions, then establish a benchmark for construction so completion can be judged by a working dense representation, usable Python translation, and measured performance.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python, rust
Domain
data, performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.