daisybio / daisybio/domainsplit

Remap dead Pfam accessions to their #=GF FW successor instead of dropping their DDIs

Open
#5 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
0
Forks
0
Avg merge
19h 30m
Merged PRs (30d)
1

Description

## Summary

Dead Pfam accessions are currently dropped. They could be **remapped** to their
successor family instead, recovering the DDIs that reference them.

`FETCH_DOMAIN_META` already reports them correctly — `dropped_families.tsv` carries
reason `dead`, and `PRUNE_UNREPRESENTED_DDIS` removes the DDIs that touch them. So
nothing is wrong today; this is recoverable data being discarded.

## The data

`Pfam-A.dead.gz` (26 KB) carries a forward pointer for **every** killed family —
measured on the current release: **1121 dead families, 1121 with `#=GF FW`**.

```
#=GF ID NtCtMGAM_N
#=GF AC PF16863
#=GF KL This family has been killed
#=GF FW PF13802
#=GF CC Merged into existing family that has been expanded to cover
#=GF CC whole structural domain
```

So `PF16863 -> PF13802` is a lookup, not a heuristic.

Dead accessions enter through the DDI sources (3did, PPIDM, Negatome), which are
built against older Pfam releases than the one `FETCH_DOMAIN_META` streams.

## Where the remap has to happen

**Downstream, on the DDI rows in the master DB, before `EXPORT_UNION_FAMILIES`.**

Remapping only the instance fetch does not work: the `ddi` rows would still be keyed
on `PF16863` while the instances come back keyed on `PF13802`, so
`PRUNE_UNREPRESENTED_DDIS` still deletes them. The rewrite must cover `ddi.pfam_a`,
`ddi.pfam_b` and `domain.pfam_id`.

## Implementation notes

- New param `url_pfam_dead` (`https://ftp.ebi.ac.uk/pub/databases/Pfam/current_release/Pfam-A.dead.gz`).
- New module, between `COLLECT_DDI_DATA` and `EXPORT_UNION_FAMILIES`.
- **Collision handling is mandatory**: `PF16863-X` and `PF13802-X` collapse into one
DDI once remapped. Needs dedupe on `(pfam_a, pfam_b, source, negative)`, and the
merge must not silently turn a positive and a negative DDI into one row.
- Forward pointers can chain (`A -> B -> C`); resolve transitively with a cycle guard.
- Report how many DDIs were remapped and how many collapsed, in
`report_ddi_attrition.py` — a remap that quietly halves a source is worse than a drop.
- Semantics caveat worth stating in the docs: `#=GF CC` usually says "merged into an
existing family that has been expanded", so the successor family is *broader* than
the one the source DDI was annotated against.

Contributor guide

Open the contributing guide

Research direction

Trace the pipeline from COLLECT_DDI_DATA through PRUNE_UNREPRESENTED_DDIS and EXPORT_UNION_FAMILIES, focusing on the master DB fields ddi.pfam_a, ddi.pfam_b, and domain.pfam_id. Read report_ddi_attrition.py and inspect how DDI rows are deduplicated and reported. Done means transitively remapped dead accessions preserve eligible DDIs, handle collisions without merging conflicting polarity, and report remapped and collapsed counts.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.