huggingface / huggingface/carbon

[Eval] Add chromatin accessibility benchmark to the zero-shot eval suite

Open
#18 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
214
Forks
27
PR merge metrics
No merged PRs in 30d

Description

Hey, I went through the eval suite carefully. It's a solid set of zero-shot tasks covering VEP, sequence recovery, perturbation sensitivity, and long-context retrieval. One notable gap: there's no chromatin accessibility eval, despite the model being trained on eukaryotic regulatory sequence where chromatin state is a primary functional signal.

A natural addition would be a zero-shot ATAC-seq peak discrimination task given two sequences from the same genomic region, one from an open chromatin peak (ENCODE ATAC-seq narrowPeak) and one from flanking closed chromatin, does the model assign higher log-likelihood to the open one? This follows the same pairwise discrimination pattern already used in the perturbation tasks (mean(LL(real) > LL(perturbed))), so it slots cleanly into the existing eval structure.
A minimal implementation would use ENCODE ATAC-seq peak calls (e.g. GM12878 or K562, already publicly available) as positive examples, with matched GC-content flanking regions as negatives. Window size could mirror the VEP setup (8 kb centered on peak summit).

I work on ATAC-seq pipelines and would be happy to put together a PR for evaluation/atac_eval.py if this direction looks right to the team.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by comparing the existing perturbation tasks and the VEP setup to understand the pairwise discrimination pattern and 8 kb windows. Then review the proposed evaluation/atac_eval.py entry point and ENCODE ATAC-seq narrowPeak data for GM12878 or K562, using matched GC-content flanking regions as negatives. Done means the suite reports whether open-chromatin sequences receive higher mean log-likelihood.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
bioinformatics, machine-learning, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.