ansible / ansible/ansible-chatbot-stack

Safety shields

Open Beginner friendly
#259 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2
Forks
17
Avg merge
2d 5h
Merged PRs (30d)
49

Description

## Summary

The `inline::llama-guard` safety provider is loaded at startup but **never invoked on any request**. The `registered_resources.shields` list is empty, which causes `client.shields.list()` to return nothing, making `run_shield_moderation()` a no-op on every query.

## Evidence

**1. No shields registered in the configmap**

```yaml
registered_resources:
shields: [] # <-- empty
```

The provider and default shield ID are declared, but without a shield entry here, nothing references the provider:

```yaml
providers:
safety:
- provider_id: llama-guard
provider_type: inline::llama-guard
config:
excluded_categories: []

registered_resources:
safety:
- default_shield_id: llama-guard
```

**2. Application-layer moderation silently passes**

`streaming_query.py` calls `run_shield_moderation()` on every request. This function discovers shields via `client.shields.list()`. With an empty list, the moderation loop never executes and returns `ShieldModerationPassed()`:

```
streaming_query.py -> retrieve_response_generator() -> run_shield_moderation()
-> client.shields.list() # returns []
-> shields_to_run = list(all_shields) # empty
-> for shield in shields_to_run: ... # never executes
-> return ShieldModerationPassed()
```

**3. No shield activity in pod logs**

Logs show only readiness probes and `POST /v1/streaming_query` requests. Zero entries for shield execution, safety violations, or guard invocations.

**4. Observed on:** prod2-west cluster, namespace `wisdom-ls-stack`, deployment `lm-stack-server` (image `ansible-chatbot-stack:0.9.202603241302`)

## How to Enable

The `inline::llama-guard` provider is a prompt-based safety classifier. It wraps user input in a structured prompt with safety categories (S1-S14), sends it to any inference model via `inference_api.chat_completion()`, and parses the response for `safe` or `unsafe\nS1,S2,...`. It does **not** require an actual Llama Guard model — any LLM that can follow the prompt format works.

### Option 1: Use the existing main inference model (all providers)

The simplest option — point the shield at whichever inference model is already registered. This works for any provider (Vertex AI, vLLM, OpenAI) since the shield just calls `inference_api.chat_completion()` with the given model ID.

```yaml
registered_resources:
shields:
- shield_id: llama-guard
provider_id: llama-guard
provider_resource_id:
```

For example, if the main model is registered as `vertexai/gemini-2.0-flash` or `my_rhoai_dev/granite-3.1`, just use that same ID. No additional model registration needed.

**Tradeoff:** Every request makes two LLM calls to the same model (one for safety, one for the actual response), which adds latency.

### Option 2: Register a dedicated smaller/faster guardrail model (recommended)

Use a lighter model for the safety check to minimize latency overhead. The guardrail model only needs to classify content, not generate full responses.

#### Vertex AI

```yaml
registered_resources:
models:
# ADD: dedicated guardrail model
- metadata: {}
model_id: vertexai/gemini-2.0-flash
provider_id: vertexai
provider_model_id: gemini-2.0-flash
shields:
- shield_id: llama-guard
provider_id: llama-guard
provider_resource_id: vertexai/gemini-2.0-flash
```

#### OpenAI

```yaml
registered_resources:
models:
- metadata: {}
model_id: openai/gpt-4o-mini
provider_id: openai
provider_model_id: gpt-4o-mini
shields:
- shield_id: llama-guard
provider_id: llama-guard
provider_resource_id: openai/gpt-4o-mini
```

### Option 3: Deploy a dedicated Llama Guard model (vLLM)

Serve `meta-llama/Llama-Guard-4-12B` (or `3-8B` for lower resource usage) on a separate vLLM endpoint, register it as an additional inference provider, and point the shield to it. This gives the best classification accuracy since the model is purpose-built for this prompt format.

### Verification

After updating the configmap and restarting:

1. Check startup logs for `"Available shields: ['llama-guard']"` instead of `"No available shields. Disabling safety"`
2. Hit the `GET /shields` endpoint to confirm the shield is listed
3. Send a test query with harmful content and verify it gets blocked

## Notes

- The guardrail model receives a structured Llama Guard prompt and must respond with `safe` or `unsafe\nS1,S2,...`. Models not fine-tuned for this format may need testing for consistency.
- `excluded_categories: []` means all 14 categories are active (S1 Violent Crimes through S14 Code Interpreter Abuse).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the configmap's registered_resources.shields entry and trace the streaming_query.py path through retrieve_response_generator() to run_shield_moderation(). Register the llama-guard shield with an available model, then restart and verify the startup log reports it, GET /shields lists it, and harmful test input is blocked.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, security
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.