anthropics / anthropics/jacobian-lens
Two easy-to-hit measurement pitfalls when reading apply() output as evidence of internal state
- Lingua principale
- Python
- Stelle
- 1.9k
- Fork
- 282
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Descrizione
While building an auditing pipeline on top of `jlens` (context: GNS-Foundation fork; see also PR #4), we hit two failure modes that we suspect will be common among users reading lens output as evidence of what a model is "thinking," and that currently nothing in the API surface warns about.
**1. Input-copying ceiling.** If a probed token appears anywhere in the prompt, the lens reads it at rank ~1 at that position — the readout reflects the input token, not workspace content. In our first pipeline version this silently saturated every measurement: erased/retained experimental conditions became identical to three decimals, at salience exactly ln(vocab_size). Anyone measuring "is concept X on the model's mind" over a prompt that mentions X will get a trivially confounded yes. *Possible mitigations:* a `positions=` example in the README/walkthrough showing window-restricted measurement, or a documented helper for "measure only positions after this prefix."
**2. Unfitted-position readouts.** `fit()` skips the first `SKIP_FIRST_N_POSITIONS` (16) positions, but `apply()` happily returns readouts at positions 0–15, where the lens is out-of-distribution. We found these readouts inflated and unstable (short prompts are the trap: a 25-token prompt gets 60% of its positions from the unfitted region). *Possible mitigations:* a warning when `apply()` is called with positions below the fitting floor, or documenting the constraint prominently.
Happy to PR either mitigation if maintainers have a preference on approach. Full pitfall ledger (six items, most experimental-design-level rather than API-level) is in our fork under [`experiments/rq1/METHODOLOGY.md`](https://github.com/GNS-Foundation/jacobian-lens/blob/grafomem/audit-module/experiments/rq1/METHODOLOGY.md).
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Direzione di ricerca
Start with the proposed `positions=` example in the README or walkthrough, then read `experiments/rq1/METHODOLOGY.md` for the full pitfall context. Inspect how `fit()` skips the first 16 positions and how `apply()` handles positions, then clarify whether documentation or an API warning is preferred. Done means both input-copying and unfitted-position risks are addressed consistently.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- python
- Ambito
- machine-learning
- Tipo di issue
- Funzionalità
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Stato di attività
- Tranquilla
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 48/100