antirez / antirez/ds4

Server: expose directional steering as a model-callable DSML tool

Abierto
#190 0 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
C
Estrellas
22.4k
Forks
2.1k
Merge medio
2 d 13 h
PR fusionados (30 d)
5

Descripción

Build on top of #148 to expose directional steering as a DSML tool the
model can invoke during its own generation, so it can decide turn-by-turn
to bias its own activations: more concise, less yapping, less refusal on
legitimate technical questions, additive concept-fixation for artistic
experiments, and so on.

PR #148 already gives the engine-side substrate — per-session steering
overrides, tool-grammar-safe `final-answer` policy, MTP gating during
dynamic steering. The remaining work proposed here is the model-facing
surface on top: a DSML tool, a vector registry with allowlist, scope
semantics, and disk-KV persistence of the active steering state.

Filed as a proposal/RFC rather than a PR because #148 is still open and
the design space is moving (see the #148 thread on conditional/additive
modes).

## Motivation

Today steering is configured at server startup
(`--dir-steering-policy`, `DS4_DIR_STEERING_FFN`, `DS4_DIR_STEERING_ATTN`).
With #148, the engine can change scales mid-session, but only an external
operator can drive that. Letting the model call `steer()` is a strictly
different capability:

- During a long agent loop, the model can request a "concise" vector for
one turn before emitting a TL;DR, then drop back to baseline.
- Before a security-research answer that the unsteered model would soft-
refuse, the model can lower a refusal vector once the legitimate context
has been established by the user — without the operator pre-configuring
it at server start.
- For artistic experiments — the "Golden Gate" mode raised in #148 — the
model can opt into additive steering on a fixed concept for the duration
of one creative task.

No other local engine exposes steering as a first-class control surface
at this granularity, let alone to the model itself. It is a uniquely DS4
move, and consistent with the README framing of steering as a first-class
citizen of the ecosystem.

## Proposed DSML tool surface

Tool name: `steer`

Parameters:

- `vector`: string — name from a server-side registry (allowlisted).
Unknown names rejected.
- `mode`: enum `{ablation, threshold, additive}`, default `ablation`
(current behavior).
- `ffn_scale`: float — clamped per-vector. Defaults to the per-vector
value from the registry.
- `attn_scale`: float — default `0`. Rejected if non-zero unless
`--allow-attn-steering` is set at server start. The empirical evidence
in the #148 thread is that nonzero attn breaks DSML grammar.
- `threshold`: float — meaningful only when `mode=threshold`, default
`0.5`. Steering applied only when
`dot(vec, activations) > threshold * |vec|^2`. Follows CAST
(Lee et al. 2024, arXiv:2409.05907).
- `scope`: enum `{next_message, until_revert, off}`. `off` clears any
active steering.
- `reason`: optional string, traced via `--trace`, not enforced.

Returns a small JSON object:

```json
{
"ok": true,
"previous": {"vector": "...", "mode": "...", "ffn": 0.0,
"attn": 0.0, "scope": "..."},
"current": {"vector": "...", "mode": "...", "ffn": 0.0,
"attn": 0.0, "scope": "..."},
"error": null
}
```

The three `mode` values cover the design space outlined in #148:

- `ablation` is the current implementation: subtract the projection along
the vector. `activations -= dot(vec, activations) * vec`.
- `threshold` follows CAST: apply only when the projection crosses a
threshold. Conservative, preserves baseline quality on out-of-domain
prompts.
- `additive` is the "Golden Gate" mode: `activations += vec * scale`.
Useful for artistic experiments and concept-fixation.

Including all three from the DSML surface day-one means later engine work
to land `threshold` and `additive` does not force a tool-signature change
and doesn't break exact-DSML replay for cached tool calls.

## Vector registry

New directory `dir-steering/vectors/` plus a sidecar
`dir-steering/vectors.json`:

```json
{
"concise": {
"file": "concise.f32",
"description": "Reduces preamble and restating tokens.",
"default_ffn": -0.5,
"max_ffn": 1.0,
"model_callable": true,
"allowed_modes": ["ablation", "threshold"]
},
"uncertainty": {
"file": "uncertainty_ablit_imatrix.f32",
"description": "Stakeholder framing on contested-sovereignty prompts.",
"default_ffn": -0.75,
"max_ffn": 1.0,
"model_callable": false,
"allowed_modes": ["ablation"]
}
}
```

Rules:

- Only entries with `model_callable: true` can be invoked from a DSML
`steer` call. Operator-only vectors stay invisible to the model.
- `additive` mode is per-vector opt-in via `allowed_modes`, since it is
the most behavior-altering.
- New CLI flags: `--steer-vector-dir PATH`,
`--steer-allowlist a,b,c` (further restrict at runtime).

## Safety

The literature contains a real concern ("The Rogue Scalpel"): even random
steering can increase compliance with harmful requests. Mitigations
encoded in the tool surface:

- Hard clamp `|ffn_scale| <= max_ffn` per vector, enforced server-side,
never trust the model.
- Reject `attn_scale != 0` unless `--allow-attn-steering`.
- Refuse `steer()` calls emitted inside an in-flight DSML tool block.
- Allowlist gating per vector and per mode (registry + CLI flag).
- Trace every `steer` call via `--trace` with the requested parameters
and the resolved scales after clamping.

The operator remains the source of truth for what is steerable. The model
gets a curated menu, not a knob into raw activation space.

## KV cache persistence

The disk KV header already uses extension flag bit 0 for the tool-id map
(KTM section). Allocate bit 1 for an appended steering-state section:

```
magic = "KSS", version = 1
u32 vector_name_len; u8 vector_name[name_len]
u8 mode (0=ablation, 1=threshold, 2=additive)
f32 ffn_scale; f32 attn_scale; f32 threshold
u8 scope (0=off, 1=next_message, 2=until_revert)
u8 reserved[3]
```

Restore on cache hit so resumed sessions inherit the active steering
state. Files written without this section remain readable; the bit is
strictly additive.

Tokens generated under steering already have steered K/V baked into the
stored attention state. Replaying a cached prefix must NOT re-apply
steering; only new tokens beyond the cache react to the current state.
Worth a comment block next to the decode loop.

## Dependencies

This proposal sits on top of #148. The per-session steering override API
introduced there is what makes a model-callable tool feasible without
re-implementing the policy machinery. The new `steer` handler is small:

1. Parse DSML parameters; validate against the registry; clamp.
2. Build a `ds4_steering_state` (new struct in `ds4.h`).
3. Call the per-session override API from #148.
4. Emit the structured JSON result.

Everything below the override API — when steering is applied during
decode, how thinking and DSML tool-call grammar are excluded, how MTP is
gated — comes for free from #148.

## Open questions

1. **Default scope.** `next_message` is the safer default (auto-revert).
`until_revert` matches "I want to keep this on" intent better. Open
to discuss.
2. **Mode ordering.** Ship `ablation` first (already implemented), add
`additive` next (cheap), defer `threshold` until CAST lands in the
engine. Or all three from day one?
3. **Composability.** Currently one active vector per session. Multi-
vector composition is a separate, larger design — out of scope here.
4. **Discovery.** Should the model see `vectors.json` content in its tool
description automatically, or should there be a sibling
`list_steer_vectors()` tool the model calls when needed?
5. **Steering during thinking.** #148 thread raised a "between decoding
and final-answer" policy that steers thinking but not tool calls.
Worth exposing as a `scope` variant?

## Acceptance criteria (if this lands as a PR target)

- `./ds4_test --logprob-vectors` passes byte-identical with steering off.
- Integration test: agent loop of 30+ tool calls with
`steer until_revert ffn=-0.75` active throughout; every DSML tool block
parses cleanly.
- Integration test: `steer next_message` shows single-turn effect on
reply length, baseline restored on the following turn.
- Unit tests: clamp behavior, unknown-vector rejection, attn rejection,
mode allowlist rejection, scope state machine across turns.
- KV persistence round-trip test (write `until_revert`, shutdown,
restart, observe restored state via `--trace`).

## References

- Arditi et al., 2024. *Refusal in Language Models Is Mediated by a
Single Direction.* arXiv:2406.11717
- Lee et al. (IBM), 2024. *Programming Refusal with Conditional
Activation Steering* (CAST). arXiv:2409.05907
- "The Rogue Scalpel" — random steering increases harmful compliance.
- #148 — server: add tool-safe directional steering policy.

/cc @audreyt — this depends on your #148 substrate, would love your read
on the mode/threshold parameterization in particular.

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.