instadeepai / instadeepai/winnow

feat: accept parquet InstaNovo predictions

Open
#244 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
11
Forks
2
Avg merge
1d 8h
Merged PRs (30d)
1

Description

## Summary
`InstaNovoDatasetLoader` currently rejects anything except .csv in `_load_predictions_without_beams` and `_load_beam_preds`. Spectrum files already support parquet/IPC/MGF. Pipelines that write InstaNovo predictions as Parquet (for example INFlow) have to convert to CSV first.

Parquet predictions will store beam and token columns as Polars list types, not as the CSV string encodings the loader is built around. Therefore `_process_beams` and `_process_predictions` must be adapted to accept those lists.

## Docs / tests
- Update `InstaNovoDatasetLoader` docstrings, docs/cli.md data formats, docs/api/datasets.md and winnow/configs/data_loader/instanovo.yaml comments.
- Rename `test_load_beam_preds_raises_for_non_csv` to reject an unsupported suffix (for example .txt), not .parquet.
- Add parquet fixtures written with real Polars list dtypes, not CSV-like strings inside parquet:
- load without beams
- `_load_beam_preds` split
- `_process_beams` → `ScoredSequence` with list `token_log_probabilities` and correct token sequences
top-hit `predictions_tokenised` as `List(Utf8)` through `_process_predictions`

### Description & Purpose:

_No response_

### Additional Notes:

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with InstaNovoDatasetLoader methods _load_predictions_without_beams, _load_beam_preds, _process_beams, and _process_predictions, then inspect the existing loader tests and CSV handling. Add real Polars-list parquet fixtures for the listed loading and processing cases, and update docs/cli.md, docs/api/datasets.md, and winnow/configs/data_loader/instanovo.yaml. Done means parquet predictions load correctly, unsupported suffixes still fail, and token sequences, probabilities, and predictions_tokenised have the required list types.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, documentation, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.