bio-learn / bio-learn/biolearn
BoAChallengeData loads an empty methylation matrix (930659 × 0 samples): parser line numbers off-by-one vs current GSE246337 series matrix
- Dominant language
- Python
- Stars
- 91
- Forks
- 39
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
`DataLibrary().get("BoAChallengeData").load()` returns a methylation matrix with **zero samples** — shape `(930659, 0)` — and scrambled metadata. The cause is that the `biomarkers-challenge-2024` parser config for this dataset uses **hardcoded series-matrix line numbers that are off by one** relative to the current `GSE246337_series_matrix.txt`, so the betas-column → GSM mapping comes out empty and every sample column is pruned.
This makes the 2024 Biomarkers-of-Aging Challenge dataset unusable as currently shipped.
## Reproduction
```python
from biolearn.data_library import DataLibrary
d = DataLibrary().get("BoAChallengeData").load()
print(d.dnam.shape) # (930659, 0) <- 0 samples
print(d.metadata.shape) # (1003, 7) <- 500 subjects, but 1003 rows
print(d.metadata[["subject_id", "tissue", "race1"]].notna().sum()) # mostly NaN / shifted
```
**Environment:** biolearn 0.9.1, Python 3.11.13, pandas 2.2.3, numpy 2.1.3 (also reproduced on pandas 3.0.3 — independent of pandas version).
## Root cause
The `BoAChallengeData` entry in `data/library.yaml` declares (abridged):
```yaml
parser:
type: biomarkers-challenge-2024
id-row: 31
matrix-file-key-line: 53
metadata:
subject_id: { row: 39 }
tissue: { row: 40 }
sex: { row: 41 }
race1: { row: 42 }
race2: { row: 43 }
ethnicity: { row: 44 }
age: { row: 45 }
```
But the **current** `GSE246337_series_matrix.txt` has these fields one row lower:
| Field | config row | actual row (current file) | content at the config row |
|---|---|---|---|
| sample key (`id-row`) | 31 | **32** (`!Sample_geo_accession`) | row 31 is `!Sample_title` |
| betas-column key (`matrix-file-key-line`) | 53 | **54** (`!Sample_description`, the sentrix IDs) | row 53 is `!Sample_scan_protocol` |
| subject_id | 39 | **40** | `!Sample_organism_ch1` ("Homo sapiens") |
| tissue | 40 | **41** | `subject id: BoA1` |
| sex | 41 | **42** | `tissue: blood` |
| race1 | 42 | **43** | `Sex: M` |
| race2 | 43 | **44** | `race1: White` |
| ethnicity | 44 | **45** | `race2: NA` |
| age | 45 | **46** | `ethnicity: Non Hispanic` |
`ChallengeDataParser.parse` builds the betas-column→GSM mapping via `build_column_mapping(file_path, matrix_file_key_line=53, id_row=31)`, then `map_and_prune_columns` drops every betas column whose renamed value isn't in that mapping. Reproducing `build_column_mapping` on the current file:
```python
import pandas as pd
def build_mapping(path, from_key_line, to_key_line):
m = pd.read_table(path, index_col=0,
skiprows=lambda x: x != from_key_line-1 and x != to_key_line-1)
rec = m.to_dict("records")[0]
return {v: k for k, v in rec.items()} if to_key_line < from_key_line else rec
# CONFIGURED (key=53, id=31):
# 1 entry: 'Standard Illumina protocol' -> 'Genomic DNA from Blood from subject BoA1'
# CORRECTED (key=54, id=32):
# 500 entries: '207700470022_R08C01' -> 'GSM7866964' (sentrix array ID -> GSM)
```
So at the configured lines the mapping is a single nonsense pair (the repeated `!Sample_scan_protocol` value collapses to one column under pandas), the betas columns (sentrix IDs like `207700470022_R08C01`) match nothing, and `map_and_prune_columns` removes all 500 → `dnam` ends up `(930659, 0)`.
The same +1 offset shifts every metadata field onto the wrong `!Sample_characteristics_ch1` line (e.g. `race1` reads `Sex: M`), and the downstream `metadata.combine_first(proteomic_metadata)` on a non-matching index is what produces the 1003 rows for 500 subjects.
## Suggested fix
Two options:
1. **Minimal:** bump each hardcoded row in the `BoAChallengeData` parser config by +1 (`id-row: 32`, `matrix-file-key-line: 54`, metadata rows 40–46). With the corrected lines the column mapping resolves to the full 500 sentrix→GSM pairs (verified above).
2. **More robust (recommended):** key the challenge/GEO-matrix parsers off the `!Sample_*` tag names (and the `characteristics_ch1` `key:` prefixes) rather than absolute line numbers, since GEO series-matrix headers shift between revisions and will re-break any pinned line numbers. This would also fix the metadata field misalignment structurally.
Happy to open a PR with option 1 (and a small regression test asserting `dnam.shape[1] == 500`) if that'd be useful.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the BoAChallengeData entry in data/library.yaml and trace ChallengeDataParser.parse through build_column_mapping and map_and_prune_columns. Update the parser configuration or agreed tag-based lookup, then add a regression check showing 500 samples and correctly aligned metadata instead of an empty methylation matrix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- bioinformatics, data
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 72/100