Proposal: Add WFGY 16-problem RAG failure map as an optional FinGPT evaluation pack

Hi, and thanks for **FinGPT**. It has become a central reference for people building LLM and NLP systems in finance.

I maintain an MIT-licensed open-source project called **WFGY** (~1.5k GitHub stars).
One of its main components is a **16-problem “ProblemMap” for RAG and LLM pipelines**, which catalogues common failure modes across:

- data ingestion and chunking
- embeddings and vector stores
- retrievers and ranking
- LLM routing and tool calls
- evaluation gaps and guardrails

ProblemMap overview:
https://github.com/onestardao/WFGY/blob/main/ProblemMap/README.md

This checklist is already referenced or integrated by several external projects, including:

- Harvard MIMS Lab: ToolUniverse (LLM robustness and RAG debugging entry)
- QCRI LLM Lab: Multimodal RAG Survey
- curated lists such as Awesome AI in Finance and AI Agents for Cybersecurity

---

### Why this matters for FinGPT

FinGPT positions itself as **open-source financial large language models** with an evaluation suite for downstream financial tasks. In realistic deployments, users often wrap FinGPT models inside RAG pipelines and agents that answer questions over reports, news, filings and time series.

In these settings, the main pain point is not just raw model quality, but understanding **why** a financial answer fails and whether the failure comes from retrieval, embedding drift, prompt design, tool routing or evaluation gaps.

The 16-problem map gives users a shared vocabulary for these failure modes and can complement FinGPT-Bench as an **optional “stress-test pack” for financial RAG workflows**.

---

### Concrete proposal

If you think this is aligned, I would be happy to:

1. Draft a short **“RAG failure modes with FinGPT”** docs page or example notebook, using the 16-problem map as the backbone, focused on financial QA over text plus simple time series context.

2. Show, on a small end-to-end example, how a few representative failure modes (for example hallucinated risk statements, misrouted tools, broken retrieval over filings) manifest in a FinGPT-based pipeline and which components users should inspect first.

3. Add a compact troubleshooting table:
*Symptom → ProblemMap number → which part of the FinGPT pipeline or surrounding stack to look at first.*

This would be a **docs-only contribution** and does not require any change to the core FinGPT models or training code. The evaluation pack can stay optional and external so that users who care about robustness can opt in.

If this sounds useful, I can open a PR with a first draft and adjust based on your feedback.