huggingface / huggingface/carbon
[Eval] Add chromatin accessibility benchmark to the zero-shot eval suite
- Dominant language
- Python
- Stars
- 214
- Forks
- 27
- PR merge metrics
- No merged PRs in 30d
Description
Hey, I went through the eval suite carefully. It's a solid set of zero-shot tasks covering VEP, sequence recovery, perturbation sensitivity, and long-context retrieval. One notable gap: there's no chromatin accessibility eval, despite the model being trained on eukaryotic regulatory sequence where chromatin state is a primary functional signal.
A natural addition would be a zero-shot ATAC-seq peak discrimination task given two sequences from the same genomic region, one from an open chromatin peak (ENCODE ATAC-seq narrowPeak) and one from flanking closed chromatin, does the model assign higher log-likelihood to the open one? This follows the same pairwise discrimination pattern already used in the perturbation tasks (mean(LL(real) > LL(perturbed))), so it slots cleanly into the existing eval structure.
A minimal implementation would use ENCODE ATAC-seq peak calls (e.g. GM12878 or K562, already publicly available) as positive examples, with matched GC-content flanking regions as negatives. Window size could mirror the VEP setup (8 kb centered on peak summit).
I work on ATAC-seq pipelines and would be happy to put together a PR for evaluation/atac_eval.py if this direction looks right to the team.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by comparing the existing perturbation tasks and the VEP setup to understand the pairwise discrimination pattern and 8 kb windows. Then review the proposed evaluation/atac_eval.py entry point and ENCODE ATAC-seq narrowPeak data for GM12878 or K562, using matched GC-content flanking regions as negatives. Done means the suite reports whether open-chromatin sequences receive higher mean log-likelihood.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- bioinformatics, machine-learning, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100