antirez / antirez/ds4

Server: expose directional steering as a model-callable DSML tool

Offen
#190 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
C
Sterne
22.4k
Forks
2.1k
Ø Merge
2 T. 13 Std.
Gemergte PRs (30 T.)
5

Beschreibung

Build on top of #148 to expose directional steering as a DSML tool the
model can invoke during its own generation, so it can decide turn-by-turn
to bias its own activations: more concise, less yapping, less refusal on
legitimate technical questions, additive concept-fixation for artistic
experiments, and so on.

PR #148 already gives the engine-side substrate — per-session steering
overrides, tool-grammar-safe `final-answer` policy, MTP gating during
dynamic steering. The remaining work proposed here is the model-facing
surface on top: a DSML tool, a vector registry with allowlist, scope
semantics, and disk-KV persistence of the active steering state.

Filed as a proposal/RFC rather than a PR because #148 is still open and
the design space is moving (see the #148 thread on conditional/additive
modes).

## Motivation

Today steering is configured at server startup
(`--dir-steering-policy`, `DS4_DIR_STEERING_FFN`, `DS4_DIR_STEERING_ATTN`).
With #148, the engine can change scales mid-session, but only an external
operator can drive that. Letting the model call `steer()` is a strictly
different capability:

- During a long agent loop, the model can request a "concise" vector for
one turn before emitting a TL;DR, then drop back to baseline.
- Before a security-research answer that the unsteered model would soft-
refuse, the model can lower a refusal vector once the legitimate context
has been established by the user — without the operator pre-configuring
it at server start.
- For artistic experiments — the "Golden Gate" mode raised in #148 — the
model can opt into additive steering on a fixed concept for the duration
of one creative task.

No other local engine exposes steering as a first-class control surface
at this granularity, let alone to the model itself. It is a uniquely DS4
move, and consistent with the README framing of steering as a first-class
citizen of the ecosystem.

## Proposed DSML tool surface

Tool name: `steer`

Parameters:

- `vector`: string — name from a server-side registry (allowlisted).
Unknown names rejected.
- `mode`: enum `{ablation, threshold, additive}`, default `ablation`
(current behavior).
- `ffn_scale`: float — clamped per-vector. Defaults to the per-vector
value from the registry.
- `attn_scale`: float — default `0`. Rejected if non-zero unless
`--allow-attn-steering` is set at server start. The empirical evidence
in the #148 thread is that nonzero attn breaks DSML grammar.
- `threshold`: float — meaningful only when `mode=threshold`, default
`0.5`. Steering applied only when
`dot(vec, activations) > threshold * |vec|^2`. Follows CAST
(Lee et al. 2024, arXiv:2409.05907).
- `scope`: enum `{next_message, until_revert, off}`. `off` clears any
active steering.
- `reason`: optional string, traced via `--trace`, not enforced.

Returns a small JSON object:

```json
{
"ok": true,
"previous": {"vector": "...", "mode": "...", "ffn": 0.0,
"attn": 0.0, "scope": "..."},
"current": {"vector": "...", "mode": "...", "ffn": 0.0,
"attn": 0.0, "scope": "..."},
"error": null
}
```

The three `mode` values cover the design space outlined in #148:

- `ablation` is the current implementation: subtract the projection along
the vector. `activations -= dot(vec, activations) * vec`.
- `threshold` follows CAST: apply only when the projection crosses a
threshold. Conservative, preserves baseline quality on out-of-domain
prompts.
- `additive` is the "Golden Gate" mode: `activations += vec * scale`.
Useful for artistic experiments and concept-fixation.

Including all three from the DSML surface day-one means later engine work
to land `threshold` and `additive` does not force a tool-signature change
and doesn't break exact-DSML replay for cached tool calls.

## Vector registry

New directory `dir-steering/vectors/` plus a sidecar
`dir-steering/vectors.json`:

```json
{
"concise": {
"file": "concise.f32",
"description": "Reduces preamble and restating tokens.",
"default_ffn": -0.5,
"max_ffn": 1.0,
"model_callable": true,
"allowed_modes": ["ablation", "threshold"]
},
"uncertainty": {
"file": "uncertainty_ablit_imatrix.f32",
"description": "Stakeholder framing on contested-sovereignty prompts.",
"default_ffn": -0.75,
"max_ffn": 1.0,
"model_callable": false,
"allowed_modes": ["ablation"]
}
}
```

Rules:

- Only entries with `model_callable: true` can be invoked from a DSML
`steer` call. Operator-only vectors stay invisible to the model.
- `additive` mode is per-vector opt-in via `allowed_modes`, since it is
the most behavior-altering.
- New CLI flags: `--steer-vector-dir PATH`,
`--steer-allowlist a,b,c` (further restrict at runtime).

## Safety

The literature contains a real concern ("The Rogue Scalpel"): even random
steering can increase compliance with harmful requests. Mitigations
encoded in the tool surface:

- Hard clamp `|ffn_scale| <= max_ffn` per vector, enforced server-side,
never trust the model.
- Reject `attn_scale != 0` unless `--allow-attn-steering`.
- Refuse `steer()` calls emitted inside an in-flight DSML tool block.
- Allowlist gating per vector and per mode (registry + CLI flag).
- Trace every `steer` call via `--trace` with the requested parameters
and the resolved scales after clamping.

The operator remains the source of truth for what is steerable. The model
gets a curated menu, not a knob into raw activation space.

## KV cache persistence

The disk KV header already uses extension flag bit 0 for the tool-id map
(KTM section). Allocate bit 1 for an appended steering-state section:

```
magic = "KSS", version = 1
u32 vector_name_len; u8 vector_name[name_len]
u8 mode (0=ablation, 1=threshold, 2=additive)
f32 ffn_scale; f32 attn_scale; f32 threshold
u8 scope (0=off, 1=next_message, 2=until_revert)
u8 reserved[3]
```

Restore on cache hit so resumed sessions inherit the active steering
state. Files written without this section remain readable; the bit is
strictly additive.

Tokens generated under steering already have steered K/V baked into the
stored attention state. Replaying a cached prefix must NOT re-apply
steering; only new tokens beyond the cache react to the current state.
Worth a comment block next to the decode loop.

## Dependencies

This proposal sits on top of #148. The per-session steering override API
introduced there is what makes a model-callable tool feasible without
re-implementing the policy machinery. The new `steer` handler is small:

1. Parse DSML parameters; validate against the registry; clamp.
2. Build a `ds4_steering_state` (new struct in `ds4.h`).
3. Call the per-session override API from #148.
4. Emit the structured JSON result.

Everything below the override API — when steering is applied during
decode, how thinking and DSML tool-call grammar are excluded, how MTP is
gated — comes for free from #148.

## Open questions

1. **Default scope.** `next_message` is the safer default (auto-revert).
`until_revert` matches "I want to keep this on" intent better. Open
to discuss.
2. **Mode ordering.** Ship `ablation` first (already implemented), add
`additive` next (cheap), defer `threshold` until CAST lands in the
engine. Or all three from day one?
3. **Composability.** Currently one active vector per session. Multi-
vector composition is a separate, larger design — out of scope here.
4. **Discovery.** Should the model see `vectors.json` content in its tool
description automatically, or should there be a sibling
`list_steer_vectors()` tool the model calls when needed?
5. **Steering during thinking.** #148 thread raised a "between decoding
and final-answer" policy that steers thinking but not tool calls.
Worth exposing as a `scope` variant?

## Acceptance criteria (if this lands as a PR target)

- `./ds4_test --logprob-vectors` passes byte-identical with steering off.
- Integration test: agent loop of 30+ tool calls with
`steer until_revert ffn=-0.75` active throughout; every DSML tool block
parses cleanly.
- Integration test: `steer next_message` shows single-turn effect on
reply length, baseline restored on the following turn.
- Unit tests: clamp behavior, unknown-vector rejection, attn rejection,
mode allowlist rejection, scope state machine across turns.
- KV persistence round-trip test (write `until_revert`, shutdown,
restart, observe restored state via `--trace`).

## References

- Arditi et al., 2024. *Refusal in Language Models Is Mediated by a
Single Direction.* arXiv:2406.11717
- Lee et al. (IBM), 2024. *Programming Refusal with Conditional
Activation Steering* (CAST). arXiv:2409.05907
- "The Rogue Scalpel" — random steering increases harmful compliance.
- #148 — server: add tool-safe directional steering policy.

/cc @audreyt — this depends on your #148 substrate, would love your read
on the mode/threshold parameterization in particular.

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.