bigbio / bigbio/hvantk

ptm: export_ptm_strata still makes 3 full passes, and its new comment overstates the cause

Open
#307 0 comments 0 reactions 0 assignees View on GitHub
enhancement psroc
Dominant language
Python
Stars
1
Forks
0
Avg merge
1d 3h
Merged PRs (30d)
40

Description

#303 replaced a per-stratum `ht.count()` in `export_ptm_strata` with a single `aggregate`, taking the function from 4 full passes to 3. Two follow-ups, one of which is a correction to the comment that PR added.

## The comment overstates the problem

The comment currently reads:

> because `annotated_ht` is lazy, every one of those jobs re-ran the whole PTM annotation pipeline just to produce a log line

That is not true for the actual caller. `hvantk/tools/ptm/ptm_cli.py` does:

```python
ht = hl.read_table(annotated_ht)
paths = export_ptm_strata(ht, output)
```

so the input is an already-materialized table on disk. Each pass re-reads and re-filters it; nothing re-runs the annotation pipeline. The documented API example (`docs_site/examples/ptm.md`) also checkpoints before calling. The comment should be corrected or the claim dropped.

## The remaining passes

Current: 1 `aggregate` + 2 `export` = 3 full scans of the table.

- `annotated_ht.checkpoint(hl.utils.new_temp_file(extension="ht"))` before the split is the fix the comment describes, and would matter for a caller that *does* pass a lazy table.
- Cheaper still for the counts specifically: `output_dir` is a local path (`os.makedirs`) and `Table.export` writes a single file, so counting lines of each exported file after the export yields the same numbers with **zero** extra Spark jobs — taking the function to 2 passes.

Neither is urgent; recording so the 4->3 improvement is not mistaken for the end of it.

Contributor guide

Open the contributing guide

Research direction

Start with export_ptm_strata and its caller in hvantk/tools/ptm/ptm_cli.py, then compare the documented usage in docs_site/examples/ptm.md. Correct the comment so it matches materialized-table behavior, and verify that exported strata counts remain unchanged while avoiding the extra count scan.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, performance
Issue type
Refactor
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.