Custom risks (#324): initial findings

Note on format: these are five separate findings, grouped into one card
while the issue format is being migrated. Each section is self-contained -
steps, expected, actual - and can be split into its own issue without rewriting.

Tested against feat/custom-risk-agent (53c38ee) on a local environment, plus direct
calls to the guardrails service where noted. Findings 1 and 5 were also confirmed
against the service independently.

A lot of this works well. The replacement warning names all
five defaults it switches off, the 25-risk cap disables the Add button, switching
custom risks off keeps them as a draft,
the panel survives the service being unreachable, and a guard that doesn't accept
custom risks rejects them with a clear message. The findings below are the
exceptions.

---

1 · A risk id matching a service default is silently replaced

A custom risk whose id matches one of the service's built-in risk names is not
used - the service substitutes its own definition of that risk. The editor accepts
the id with no warning and keeps displaying the user's text, so nothing indicates
the rule isn't running.

How to see it without relying on a verdict

The response says which definition ran: the criteria it returns are either the
ones you supplied, or the service's own. Six calls to POST /guardrail/risk_agent,
identical except the id, with this rule each time:

risks:
- id: <VARIES>
  description: Acreage must be reported in hectares.
  criteria:
  - All area figures are given in hectares.

against content: "The Palisades Fire has burned 1,247 acres and is 12% contained as of 14:00 UTC."

Result — first criterion returned in each response

id sent	Criterion that came back
hallucination-identification	"No statement in the content contradicts the provided source material…"
attribution	"Claims the content presents as drawn from a specific source, dataset, or citation…"
consistency	"The content contains no mutually contradictory statements."
uncertainty-identification	"Genuinely uncertain, preliminary, or contested claims are appropriately hedged…"
overgeneralization	"The content does not restate a quantified or qualified finding as an unqualified universal claim…"
unit-convention (control)	"Acreage is reported using hectares instead of acres."

The five colliding ids returned the service's built-in criteria. Not one
mentions hectares. The control returned the rule that was sent, and failed the
content correctly - so the rule itself is valid and does fire.

Why this is substitution rather than the judge being steered by the name

The judge regenerates its rubric on every call, so a genuinely custom risk comes
back worded differently each time. Repeating the calls shows exactly that:

id	run 1	run 2
unit-convention (custom)	"Acreage is reported using hectares instead of acres." + 2 more	"Output reports acreage in hectares (not acres)" — 1 criterion
hallucination-identification (colliding)	"No statement in the content contradicts the provided source material. Not applicable when no source material is provided — a bare question or instruction is not source material."	identical, word for word
attribution (colliding)	three criteria	identical, word for word

The custom rule's criteria drift between runs, as generated rubrics do. The
colliding ids return byte-identical text — which is what a stored definition looks
like, not a generated one.

Expected — a colliding id is rejected, or the user's definition is used.

Actual — the user's definition is discarded silently and the built-in runs
instead.

---

2 · Typed YAML is discarded when switching editor tabs

Text typed into Paste YAML is lost when switching to Form and back, unless
Apply was clicked first. No warning.

Steps

1. Agent editor → Runtime → Custom Risks on → Paste YAML
2. Type or paste a risks document
3. Click Form — without clicking Apply
4. Click Paste YAML again

Expected — the typed document is still there, or a warning before it's
discarded.

Actual — the box is repopulated from the current form rows, or with the
default example if there are none. The typed text is gone.

---

3 · Validation errors are shown as raw framework output

Some save errors surface as a raw Pydantic ValidationError, including the
internal model name, a truncated dump of the submitted payload, and a link to the
Pydantic docs.

Steps

1. Agent editor → Runtime → Custom Risks on
2. Add two risks with the same id in Form mode — no warning while editing
3. Save

Expected — a readable message naming the problem, as already happens
elsewhere on this form.

Actual

Notice
Invalid guardrails: 1 validation error for AgentGuardrails
custom_risks
  Value error, duplicate custom risk id 'duplicate-test' [type=value_error,
  input_value={'enabled': True, 'risks'…er mentions a date.']}]}, input_type=dict]
  For further information visit https://errors.pydantic.dev/2.12/v/value_error

The good path already exists — saving with custom risks on and zero risks
gives:

"Invalid guardrails.custom_risks: add at least one risk or turn custom risks off."

Also — Paste YAML validates duplicates properly on Apply
(risks[1].id "duplicate-test" is duplicated.) while Form mode gives no
warning until save. The same rule is enforced in one editor and not the other.

---

4 · Contradictory rules are accepted with no warning, and behave unpredictably

Two rules that cannot both be satisfied are accepted without comment, and the
resulting agent behaves differently from run to run.

Steps

1. Add two rules requiring the same figure in different units:
   • metric-units — All area figures are given in hectares.
   • imperial-units — All area figures are given in acres.
2. Save — no warning appears
3. Ask, new conversation each time:
   What is the area of Central Park? Answer in acres, with the unit.
4. Repeat three times

Expected - unclear, and worth discussing. The rules conflict, so arguably this is user error — but nothing in the editor indicates that, and the agent's behaviour varies rather than failing consistently.

Actual — four runs of the same question produced three different outcomes:

Run	Outcome
1	Passed — answer shown, neither rule fired
2	Blocked — both rules flagged
3	Blocked — metric-units only
4 (asked in hectares)	Blocked — imperial-units only

In run 2 the report restated imperial-units as requiring hectares — the
opposite of its criterion. Reports also add conditions the rule didn't contain:
one restated a hectares rule as additionally requiring a conversion factor.

When it does block, the report suggests using the other unit — which then trips
the other rule. Nothing indicates the two rules conflict.

Worth discussing rather than simply fixing — detecting unsatisfiable rule sets
in general isn't realistic. The symptom may be more tractable: an agent whose
guard flags a high proportion of turns is a state worth surfacing to its owner.

---

5 · The editor allows more risks than the service accepts

An agent can be saved with 25 custom risks. The service rejects more than 20, so
every turn afterwards goes unscreened.

Steps

1. Create an agent, Custom Risks on
2. Add 25 risks (Paste YAML is quickest), Save — accepted, no error, the card
   reads "25 DEFINED"
3. Chat with the agent, send any message
4. Read the guardrail notice on the turn

Expected — either the editor refuses more risks than the service accepts, or
the save is rejected with a message explaining the limit.

Actual — saved with no error. Every turn showed:

"Not checked — Risk Agent did not respond (http_400), so this turn was not
screened. Retry the message to check it."

Sending the same document to the service directly returns:

{"detail": "invalid risk_yaml: too many risks: 25 (max 20)"}

The same document cut to 15 risks was accepted and judged normally.

	Limit
akd-labs — schemas.py, CustomRisks.risks	max_length=25
guardrails service	20

Suggestion — change max_length to 20, or read the limit from the service if
it can expose one.

---

Environment

• feat/custom-risk-agent (53c38ee), local stack, openai:gpt-5.2
• Guardrails service reachable throughout; findings 1 and 5 confirmed by calling
  it directly
• Findings 1, 4 and 5 each have a control run showing the expected behaviour in
  the non-failing case
• Screenshots and full JSON responses are available if needed

---

Start by separating the five findings into focused issues, then inspect akd-labs schemas.py and the Agent editor paths under Runtime and Custom Risks. Reproduce the documented requests to POST /guardrail/risk_agent and the Form/Paste YAML flows, and use the stated expected behavior and service limits to verify each fix.