anthropics / anthropics/jacobian-lens

Two easy-to-hit measurement pitfalls when reading apply() output as evidence of internal state

Offen
#5 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Python
Sterne
1.9k
Forks
282
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

While building an auditing pipeline on top of `jlens` (context: GNS-Foundation fork; see also PR #4), we hit two failure modes that we suspect will be common among users reading lens output as evidence of what a model is "thinking," and that currently nothing in the API surface warns about.

**1. Input-copying ceiling.** If a probed token appears anywhere in the prompt, the lens reads it at rank ~1 at that position — the readout reflects the input token, not workspace content. In our first pipeline version this silently saturated every measurement: erased/retained experimental conditions became identical to three decimals, at salience exactly ln(vocab_size). Anyone measuring "is concept X on the model's mind" over a prompt that mentions X will get a trivially confounded yes. *Possible mitigations:* a `positions=` example in the README/walkthrough showing window-restricted measurement, or a documented helper for "measure only positions after this prefix."

**2. Unfitted-position readouts.** `fit()` skips the first `SKIP_FIRST_N_POSITIONS` (16) positions, but `apply()` happily returns readouts at positions 0–15, where the lens is out-of-distribution. We found these readouts inflated and unstable (short prompts are the trap: a 25-token prompt gets 60% of its positions from the unfitted region). *Possible mitigations:* a warning when `apply()` is called with positions below the fitting floor, or documenting the constraint prominently.

Happy to PR either mitigation if maintainers have a preference on approach. Full pitfall ledger (six items, most experimental-design-level rather than API-level) is in our fork under [`experiments/rq1/METHODOLOGY.md`](https://github.com/GNS-Foundation/jacobian-lens/blob/grafomem/audit-module/experiments/rq1/METHODOLOGY.md).

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Rechercherichtung

Start with the proposed `positions=` example in the README or walkthrough, then read `experiments/rq1/METHODOLOGY.md` for the full pitfall context. Inspect how `fit()` skips the first 16 positions and how `apply()` handles positions, then clarify whether documentation or an API warning is preferred. Done means both input-copying and unfitted-position risks are addressed consistently.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
python
Bereich
machine-learning
Issue-Typ
Feature
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Ruhig
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
48/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.