instadeepai / instadeepai/winnow
feat: accept parquet InstaNovo predictions
- Dominant language
- Python
- Stars
- 11
- Forks
- 2
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 1
Description
## Summary
`InstaNovoDatasetLoader` currently rejects anything except .csv in `_load_predictions_without_beams` and `_load_beam_preds`. Spectrum files already support parquet/IPC/MGF. Pipelines that write InstaNovo predictions as Parquet (for example INFlow) have to convert to CSV first.
Parquet predictions will store beam and token columns as Polars list types, not as the CSV string encodings the loader is built around. Therefore `_process_beams` and `_process_predictions` must be adapted to accept those lists.
## Docs / tests
- Update `InstaNovoDatasetLoader` docstrings, docs/cli.md data formats, docs/api/datasets.md and winnow/configs/data_loader/instanovo.yaml comments.
- Rename `test_load_beam_preds_raises_for_non_csv` to reject an unsupported suffix (for example .txt), not .parquet.
- Add parquet fixtures written with real Polars list dtypes, not CSV-like strings inside parquet:
- load without beams
- `_load_beam_preds` split
- `_process_beams` → `ScoredSequence` with list `token_log_probabilities` and correct token sequences
top-hit `predictions_tokenised` as `List(Utf8)` through `_process_predictions`
### Description & Purpose:
_No response_
### Additional Notes:
_No response_
Contributor guide
Research direction
Start with InstaNovoDatasetLoader methods _load_predictions_without_beams, _load_beam_preds, _process_beams, and _process_predictions, then inspect the existing loader tests and CSV handling. Add real Polars-list parquet fixtures for the listed loading and processing cases, and update docs/cli.md, docs/api/datasets.md, and winnow/configs/data_loader/instanovo.yaml. Done means parquet predictions load correctly, unsupported suffixes still fail, and token sequences, probabilities, and predictions_tokenised have the required list types.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, documentation, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100