daisybio / daisybio/domainsplit
Remap dead Pfam accessions to their #=GF FW successor instead of dropping their DDIs
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- Avg merge
- 19h 30m
- Merged PRs (30d)
- 1
Description
## Summary
Dead Pfam accessions are currently dropped. They could be **remapped** to their
successor family instead, recovering the DDIs that reference them.
`FETCH_DOMAIN_META` already reports them correctly — `dropped_families.tsv` carries
reason `dead`, and `PRUNE_UNREPRESENTED_DDIS` removes the DDIs that touch them. So
nothing is wrong today; this is recoverable data being discarded.
## The data
`Pfam-A.dead.gz` (26 KB) carries a forward pointer for **every** killed family —
measured on the current release: **1121 dead families, 1121 with `#=GF FW`**.
```
#=GF ID NtCtMGAM_N
#=GF AC PF16863
#=GF KL This family has been killed
#=GF FW PF13802
#=GF CC Merged into existing family that has been expanded to cover
#=GF CC whole structural domain
```
So `PF16863 -> PF13802` is a lookup, not a heuristic.
Dead accessions enter through the DDI sources (3did, PPIDM, Negatome), which are
built against older Pfam releases than the one `FETCH_DOMAIN_META` streams.
## Where the remap has to happen
**Downstream, on the DDI rows in the master DB, before `EXPORT_UNION_FAMILIES`.**
Remapping only the instance fetch does not work: the `ddi` rows would still be keyed
on `PF16863` while the instances come back keyed on `PF13802`, so
`PRUNE_UNREPRESENTED_DDIS` still deletes them. The rewrite must cover `ddi.pfam_a`,
`ddi.pfam_b` and `domain.pfam_id`.
## Implementation notes
- New param `url_pfam_dead` (`https://ftp.ebi.ac.uk/pub/databases/Pfam/current_release/Pfam-A.dead.gz`).
- New module, between `COLLECT_DDI_DATA` and `EXPORT_UNION_FAMILIES`.
- **Collision handling is mandatory**: `PF16863-X` and `PF13802-X` collapse into one
DDI once remapped. Needs dedupe on `(pfam_a, pfam_b, source, negative)`, and the
merge must not silently turn a positive and a negative DDI into one row.
- Forward pointers can chain (`A -> B -> C`); resolve transitively with a cycle guard.
- Report how many DDIs were remapped and how many collapsed, in
`report_ddi_attrition.py` — a remap that quietly halves a source is worse than a drop.
- Semantics caveat worth stating in the docs: `#=GF CC` usually says "merged into an
existing family that has been expanded", so the successor family is *broader* than
the one the source DDI was annotated against.
Contributor guide
Research direction
Trace the pipeline from COLLECT_DDI_DATA through PRUNE_UNREPRESENTED_DDIS and EXPORT_UNION_FAMILIES, focusing on the master DB fields ddi.pfam_a, ddi.pfam_b, and domain.pfam_id. Read report_ddi_attrition.py and inspect how DDI rows are deduplicated and reported. Done means transitively remapped dead accessions preserve eligible DDIs, handle collisions without merging conflicting polarity, and report remapped and collapsed counts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100