ProjectTech4DevAI / ProjectTech4DevAI/kaapi-guardrails
Guardrails: Topic relevance evaluation
Open
@AkhileshNegi is already working on this.
Since Apr 30, 2026.
- Dominant language
- Python
- Stars
- 5
- Forks
- 3
- Avg merge
- 9d 13h
- Merged PRs (30d)
- 2
Description
Is your feature request related to a problem?
We need to ensure that the assistant provides relevant responses in SNEHA's program domains and avoids off-topic queries. This is crucial for maintaining the quality of interactions in real-world scenarios.
Describe the solution you'd like
- Run the topic relevance guardrail from
kaapi-guardrailson a randomly sampled subset of ~14,605 conversations of SNEHA. - Start with 100 conversations to refine the prompt, then 500, and finally ~1.5k (~10% sampling).
- Document results since we don't yet have annotations or a known on-topic/off-topic ratio to stratify on.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.