anthropics / anthropics/jacobian-lens

Two easy-to-hit measurement pitfalls when reading apply() output as evidence of internal state

Aperta
#5 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Python
Stelle
1.9k
Fork
282
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

While building an auditing pipeline on top of `jlens` (context: GNS-Foundation fork; see also PR #4), we hit two failure modes that we suspect will be common among users reading lens output as evidence of what a model is "thinking," and that currently nothing in the API surface warns about.

**1. Input-copying ceiling.** If a probed token appears anywhere in the prompt, the lens reads it at rank ~1 at that position — the readout reflects the input token, not workspace content. In our first pipeline version this silently saturated every measurement: erased/retained experimental conditions became identical to three decimals, at salience exactly ln(vocab_size). Anyone measuring "is concept X on the model's mind" over a prompt that mentions X will get a trivially confounded yes. *Possible mitigations:* a `positions=` example in the README/walkthrough showing window-restricted measurement, or a documented helper for "measure only positions after this prefix."

**2. Unfitted-position readouts.** `fit()` skips the first `SKIP_FIRST_N_POSITIONS` (16) positions, but `apply()` happily returns readouts at positions 0–15, where the lens is out-of-distribution. We found these readouts inflated and unstable (short prompts are the trap: a 25-token prompt gets 60% of its positions from the unfitted region). *Possible mitigations:* a warning when `apply()` is called with positions below the fitting floor, or documenting the constraint prominently.

Happy to PR either mitigation if maintainers have a preference on approach. Full pitfall ledger (six items, most experimental-design-level rather than API-level) is in our fork under [`experiments/rq1/METHODOLOGY.md`](https://github.com/GNS-Foundation/jacobian-lens/blob/grafomem/audit-module/experiments/rq1/METHODOLOGY.md).

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Direzione di ricerca

Start with the proposed `positions=` example in the README or walkthrough, then read `experiments/rq1/METHODOLOGY.md` for the full pitfall context. Inspect how `fit()` skips the first 16 positions and how `apply()` handles positions, then clarify whether documentation or an API warning is preferred. Done means both input-copying and unfitted-position risks are addressed consistently.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
python
Ambito
machine-learning
Tipo di issue
Funzionalità
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Tranquilla
Chiarezza
Abbastanza chiara
Idoneità per principianti
48/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.