ptm: export_ptm_strata still makes 3 full passes, and its new comment overstates the cause
- Dominant language
- Python
- Stars
- 1
- Forks
- 0
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 40
Description
#303 replaced a per-stratum `ht.count()` in `export_ptm_strata` with a single `aggregate`, taking the function from 4 full passes to 3. Two follow-ups, one of which is a correction to the comment that PR added.
## The comment overstates the problem
The comment currently reads:
> because `annotated_ht` is lazy, every one of those jobs re-ran the whole PTM annotation pipeline just to produce a log line
That is not true for the actual caller. `hvantk/tools/ptm/ptm_cli.py` does:
```python
ht = hl.read_table(annotated_ht)
paths = export_ptm_strata(ht, output)
```
so the input is an already-materialized table on disk. Each pass re-reads and re-filters it; nothing re-runs the annotation pipeline. The documented API example (`docs_site/examples/ptm.md`) also checkpoints before calling. The comment should be corrected or the claim dropped.
## The remaining passes
Current: 1 `aggregate` + 2 `export` = 3 full scans of the table.
- `annotated_ht.checkpoint(hl.utils.new_temp_file(extension="ht"))` before the split is the fix the comment describes, and would matter for a caller that *does* pass a lazy table.
- Cheaper still for the counts specifically: `output_dir` is a local path (`os.makedirs`) and `Table.export` writes a single file, so counting lines of each exported file after the export yields the same numbers with **zero** extra Spark jobs — taking the function to 2 passes.
Neither is urgent; recording so the 4->3 improvement is not mistaken for the end of it.
Contributor guide
Research direction
Start with export_ptm_strata and its caller in hvantk/tools/ptm/ptm_cli.py, then compare the documented usage in docs_site/examples/ptm.md. Correct the comment so it matches materialized-table behavior, and verify that exported strata counts remain unchanged while avoiding the extra count scan.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, performance
- Issue type
- Refactor
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100