daisybio / daisybio/domainsplit

instance_tier: implement all_species_any_review_status (full TrEMBL universe)

Open
#4 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
0
Forks
0
Avg merge
19h 30m
Merged PRs (30d)
1

Description

## Summary

`instance_tier` will gain four explicit values:

| value | strata filled | status |
|---|---|---|
| `human_reviewed` | human Swiss-Prot | implemented |
| `human_any_review_status` | + human TrEMBL | being implemented |
| `all_species_reviewed` | + non-human Swiss-Prot | being implemented |
| `all_species_any_review_status` | + all TrEMBL | **not implemented — this issue** |

The first three fit the current architecture. The fourth does not, and the reason is
worth writing down so the next attempt does not start by trying it the easy way.

## Why the fourth does not fit

`ppi-splitting-pipeline`'s `bin/fetch_domains.py` builds the protein universe by
parsing a UniProt flat file fully into RAM:

```python
universe = parse_uniprot_dat(iter_gzip_lines(fh), taxa_filter) # {acc: (taxon, seq)}
```

then streams `Pfam-A.regions.tsv.gz` and looks each region's parent up in that dict.

Sizes (UniProt release 2026_03):

| file | size (gz) | entries |
|---|---|---|
| `uniprot_sprot.dat.gz` | 667 M | ~573 k |
| `uniprot_trembl.dat.gz` | **110 G** | ~250 M |

`all_species_reviewed` holds ~573 k proteins — roughly 1 GB of Python strings, which
works. `all_species_any_review_status` would hold ~250 M, which is hundreds of GB. It
is not a tuning problem.

`content_digest()` is a second, independent wall: it sorts and joins every accession
in the universe to build the cache key (0.3 s at 573 k). At 250 M that is minutes of
CPU and tens of GB of transient string.

## What implementing it actually requires

Not a bigger machine — a different join strategy. Sketch:

1. **Build a cached accession index over the full TrEMBL DAT**, once per UniProt
release: `accession -> (taxon_id, reviewed, byte_offset)`. ~250 M rows; on disk
(SQLite / LMDB / sorted binary), not in RAM. This is the expensive step (~110 GB
streamed once) and it is exactly what the `interpro_cache` is for.
2. **Tier-assign during the regions pass using that index**, so stratification stays
correct — a family that has a human member must still get it, which a
"sample first, resolve taxon later" shortcut cannot guarantee.
3. **Fetch sequences only for the sampled instances**, by seeking to the recorded
offsets. Sampled set is bounded by `ddi_examples_target * ddi_examples_pool_factor *
n_families` (currently 25 x ~20 k = ~500 k), so this stays small.
4. **Replace `content_digest()`** with an order-independent accumulator (XOR/sum of
per-accession hashes) or with a release-string + file-size digest.
5. Bump `FORMAT_VERSION` again.

Disk budget is not the constraint (cache is provisioned at 1 TB; 110 GB TrEMBL +
4.7 GB regions + a ~150 GB STRING working set still fits).

## Where the code lives

- Instance sampling and the universe: `bin/fetch_domains.py` in
`bionetslab/ppi-splitting-pipeline` (vendored here as
`subworkflows/external/ppi-splitting`, read-only).
- The `instance_tier` / `uniprot_dat_urls` parameter surface, `bin/parse_swissprot_dat.py`
and the workflow guards: this repo.

Both halves are needed: `parse_swissprot_dat.py` also parses the flat files, to fill
`protein.sequence`, `protein_go_terms` and the STRING id map, and has the same
streaming problem at full-TrEMBL scale.

Contributor guide

Open the contributing guide

Research direction

Read bin/fetch_domains.py in the vendored ppi-splitting pipeline, then inspect parse_swissprot_dat.py and the workflow guards in this repository. Trace how the instance_tier and uniprot_dat_urls parameters flow through both parsing paths. Done means full TrEMBL can be handled without loading the universe into RAM, tier assignment remains correct, sampled sequences are fetched, and content_digest plus FORMAT_VERSION are updated.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.