EleutherAI / EleutherAI/bergson

Allow a retrain bank to be built without precomputed scores

Open
#446 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
86
Forks
29
Avg merge
1d 22h
Merged PRs (30d)
37

Description

Building a bank currently requires finished attribution scores, because the validate path calls `load_scores_loss_signed` unconditionally and writes a `score_sum` column for every row of `validation.csv` — but when `exclude_zero_scores` is false, scores are only used for the document count and that reporting column, both of which are avoidable. Making scores optional there would let a filter's random controls be trained during the scoring pass instead of after it, which is about 10 hours on a 1M-document row.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the validate path that unconditionally calls load_scores_loss_signed and writes score_sum to validation.csv. Check how exclude_zero_scores controls score use, then make the no-exclusion path work without precomputed scores while preserving score-dependent behavior when exclusion is enabled; done means a retrain bank can be built during the scoring pass without requiring finished attribution scores.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, machine-learning
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
70/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.