instadeepai / instadeepai/winnow

feat: merged predict table with CSV, TSV or Parquet output

Open
#245 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
11
Forks
2
Avg merge
1d 8h
Merged PRs (30d)
1

Description

## Summary
`winnow predict` always writes two CSVs (`metadata.csv` and `preds_and_fdr_metrics.csv`). Downstream tools (including INFlow) often want a single PSM table, and parquet/TSV for I/O. Split CSV should remain the default.

## Current behaviour
`_save_predict_outputs` splits columns then writes CSV only. `CalibrationDataset.save_metadata` already knows .csv and .parquet but predict does not use it for TSV or a merged table.

## Proposed config (`predict.yaml`)
Keep `output_folder`. Add two knobs:
```{yaml}
# split (default): metadata + preds_and_fdr_metrics
# merged: one table with metadata, predictions and FDR columns
output_layout: split

# csv (default) | tsv | parquet
output_file_format: csv
```
File names (same stems, suffix from format):
| Layout | Files |
| --- | --- |
| `split` | `metadata.{csv,tsv,parquet}`, `preds_and_fdr_metrics.{csv,tsv,parquet}` |
| `merged` | `calibrated_psms.{csv,tsv,parquet}` |

CSV: comma, header, no index (as today). TSV: tab. Parquet: `format_metadata_for_export()` then `to_parquet` so nested `mz_array` / `intensity_array` stay arrays rather than stringified lists.

## Behaviour
- Default `split` + `csv` is bit-compatible with current outputs (same columns, names and CSV dialect).
- `merged` is the pre-split table: spectrum metadata, features, prediction and FDR columns, still FDR-filtered.
- Invalid `output_layout` / `output_file_format` → clear `ValueError`.
- Nested spectrum arrays: CSV/TSV may stay stringified (current export); parquet should keep native lists.

## Docs/tests
- docs/cli.md prediction output, docs/configuration.md, comments in predict.yaml.
- Unit tests on `_save_predict_outputs` (or a thin writer helper): split vs merged; csv/tsv/parquet; default path still produces the two current CSVs; merged contains `spectrum_id`, `prediction`, confidence, FDR columns and a metadata column such as `precursor_mz`.

### Description & Purpose:

_No response_

### Additional Notes:

_No response_

Contributor guide

Open the contributing guide

Research direction

Start at `_save_predict_outputs` and the `predict.yaml` configuration comments; compare its current split CSV behavior with `CalibrationDataset.save_metadata` and `format_metadata_for_export()`. Add or update unit tests for split and merged layouts across CSV, TSV, and Parquet, then update `docs/cli.md` and `docs/configuration.md`. Done means the default remains bit-compatible and each requested layout and format produces the documented files and columns.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, data, documentation, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.