bio-learn / bio-learn/biolearn

BoAChallengeData loads an empty methylation matrix (930659 × 0 samples): parser line numbers off-by-one vs current GSE246337 series matrix

Open Beginner friendly
#208 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
91
Forks
39
PR merge metrics
No merged PRs in 30d

Description

## Summary

`DataLibrary().get("BoAChallengeData").load()` returns a methylation matrix with **zero samples** — shape `(930659, 0)` — and scrambled metadata. The cause is that the `biomarkers-challenge-2024` parser config for this dataset uses **hardcoded series-matrix line numbers that are off by one** relative to the current `GSE246337_series_matrix.txt`, so the betas-column → GSM mapping comes out empty and every sample column is pruned.

This makes the 2024 Biomarkers-of-Aging Challenge dataset unusable as currently shipped.

## Reproduction

```python
from biolearn.data_library import DataLibrary
d = DataLibrary().get("BoAChallengeData").load()
print(d.dnam.shape) # (930659, 0) <- 0 samples
print(d.metadata.shape) # (1003, 7) <- 500 subjects, but 1003 rows
print(d.metadata[["subject_id", "tissue", "race1"]].notna().sum()) # mostly NaN / shifted
```

**Environment:** biolearn 0.9.1, Python 3.11.13, pandas 2.2.3, numpy 2.1.3 (also reproduced on pandas 3.0.3 — independent of pandas version).

## Root cause

The `BoAChallengeData` entry in `data/library.yaml` declares (abridged):

```yaml
parser:
type: biomarkers-challenge-2024
id-row: 31
matrix-file-key-line: 53
metadata:
subject_id: { row: 39 }
tissue: { row: 40 }
sex: { row: 41 }
race1: { row: 42 }
race2: { row: 43 }
ethnicity: { row: 44 }
age: { row: 45 }
```

But the **current** `GSE246337_series_matrix.txt` has these fields one row lower:

| Field | config row | actual row (current file) | content at the config row |
|---|---|---|---|
| sample key (`id-row`) | 31 | **32** (`!Sample_geo_accession`) | row 31 is `!Sample_title` |
| betas-column key (`matrix-file-key-line`) | 53 | **54** (`!Sample_description`, the sentrix IDs) | row 53 is `!Sample_scan_protocol` |
| subject_id | 39 | **40** | `!Sample_organism_ch1` ("Homo sapiens") |
| tissue | 40 | **41** | `subject id: BoA1` |
| sex | 41 | **42** | `tissue: blood` |
| race1 | 42 | **43** | `Sex: M` |
| race2 | 43 | **44** | `race1: White` |
| ethnicity | 44 | **45** | `race2: NA` |
| age | 45 | **46** | `ethnicity: Non Hispanic` |

`ChallengeDataParser.parse` builds the betas-column→GSM mapping via `build_column_mapping(file_path, matrix_file_key_line=53, id_row=31)`, then `map_and_prune_columns` drops every betas column whose renamed value isn't in that mapping. Reproducing `build_column_mapping` on the current file:

```python
import pandas as pd
def build_mapping(path, from_key_line, to_key_line):
m = pd.read_table(path, index_col=0,
skiprows=lambda x: x != from_key_line-1 and x != to_key_line-1)
rec = m.to_dict("records")[0]
return {v: k for k, v in rec.items()} if to_key_line < from_key_line else rec

# CONFIGURED (key=53, id=31):
# 1 entry: 'Standard Illumina protocol' -> 'Genomic DNA from Blood from subject BoA1'
# CORRECTED (key=54, id=32):
# 500 entries: '207700470022_R08C01' -> 'GSM7866964' (sentrix array ID -> GSM)
```

So at the configured lines the mapping is a single nonsense pair (the repeated `!Sample_scan_protocol` value collapses to one column under pandas), the betas columns (sentrix IDs like `207700470022_R08C01`) match nothing, and `map_and_prune_columns` removes all 500 → `dnam` ends up `(930659, 0)`.

The same +1 offset shifts every metadata field onto the wrong `!Sample_characteristics_ch1` line (e.g. `race1` reads `Sex: M`), and the downstream `metadata.combine_first(proteomic_metadata)` on a non-matching index is what produces the 1003 rows for 500 subjects.

## Suggested fix

Two options:

1. **Minimal:** bump each hardcoded row in the `BoAChallengeData` parser config by +1 (`id-row: 32`, `matrix-file-key-line: 54`, metadata rows 40–46). With the corrected lines the column mapping resolves to the full 500 sentrix→GSM pairs (verified above).
2. **More robust (recommended):** key the challenge/GEO-matrix parsers off the `!Sample_*` tag names (and the `characteristics_ch1` `key:` prefixes) rather than absolute line numbers, since GEO series-matrix headers shift between revisions and will re-break any pinned line numbers. This would also fix the metadata field misalignment structurally.

Happy to open a PR with option 1 (and a small regression test asserting `dnam.shape[1] == 500`) if that'd be useful.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the BoAChallengeData entry in data/library.yaml and trace ChallengeDataParser.parse through build_column_mapping and map_and_prune_columns. Update the parser configuration or agreed tag-based lookup, then add a regression check showing 500 samples and correctly aligned metadata instead of an empty methylation matrix.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
bioinformatics, data
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.