anthropics / anthropics/jacobian-lens
Two easy-to-hit measurement pitfalls when reading apply() output as evidence of internal state
- Dominant language
- Python
- Stars
- 1.9k
- Forks
- 282
- PR merge metrics
- No merged PRs in 30d
Description
While building an auditing pipeline on top of `jlens` (context: GNS-Foundation fork; see also PR #4), we hit two failure modes that we suspect will be common among users reading lens output as evidence of what a model is "thinking," and that currently nothing in the API surface warns about.
**1. Input-copying ceiling.** If a probed token appears anywhere in the prompt, the lens reads it at rank ~1 at that position — the readout reflects the input token, not workspace content. In our first pipeline version this silently saturated every measurement: erased/retained experimental conditions became identical to three decimals, at salience exactly ln(vocab_size). Anyone measuring "is concept X on the model's mind" over a prompt that mentions X will get a trivially confounded yes. *Possible mitigations:* a `positions=` example in the README/walkthrough showing window-restricted measurement, or a documented helper for "measure only positions after this prefix."
**2. Unfitted-position readouts.** `fit()` skips the first `SKIP_FIRST_N_POSITIONS` (16) positions, but `apply()` happily returns readouts at positions 0–15, where the lens is out-of-distribution. We found these readouts inflated and unstable (short prompts are the trap: a 25-token prompt gets 60% of its positions from the unfitted region). *Possible mitigations:* a warning when `apply()` is called with positions below the fitting floor, or documenting the constraint prominently.
Happy to PR either mitigation if maintainers have a preference on approach. Full pitfall ledger (six items, most experimental-design-level rather than API-level) is in our fork under [`experiments/rq1/METHODOLOGY.md`](https://github.com/GNS-Foundation/jacobian-lens/blob/grafomem/audit-module/experiments/rq1/METHODOLOGY.md).
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.