Proposal: CI-friendly RAG checks using WFGY 16-problem ProblemMap

Hi, and thanks for CML – CI-native ML checks are exactly what a lot of teams need.

I maintain an open-source project called **WFGY** (MIT-licensed, ~1.5k GitHub stars). A key part of it is a **16-problem “ProblemMap”** that lists common **RAG / LLM failure modes** such as retriever drift, vector store fragmentation, bad chunking strategies, and unstable prompts:

- WFGY ProblemMap (16 failure modes for RAG & LLM pipelines):
https://github.com/onestardao/WFGY/blob/main/ProblemMap/README.md

This ProblemMap has been integrated or referenced in:

- **ToolUniverse** by Harvard MIMS Lab
- **Multimodal RAG Survey** by QCRI LLM Lab

They use it mainly as a practical failure taxonomy.

### Proposal

Given that CML is already about automated checks in CI, I would like to propose an example that:

1. Runs a small RAG evaluation job in CI using CML
2. Classifies failures according to the 16 ProblemMap categories
3. Fails the CI job or surfaces warnings when specific failure-mode thresholds are exceeded

The idea is to stay fully within the CML + GitHub/GitLab workflow, and just use WFGY ProblemMap as an external, MIT-licensed checklist for naming the categories.

### Question

Would you be open to:

- A docs example or template repo showing “RAG CI checks with CML + WFGY ProblemMap”, and
- A short reference link to the ProblemMap as the failure-mode taxonomy?

If yes, I can propose a minimal example and iterate based on your feedback.

If this is not what you want to cover with CML, no problem – thank you for considering it.