pathwaycom / pathwaycom/pathway
[Proposal] Example: streaming RAG diagnostics with the WFGY 16-problem map in Pathway
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 62.3k
- Forks
- 1.7k
- PR merge metrics
- No merged PRs in 30d
Description
Is your feature request related to a problem? Please describe.
Many Pathway users build RAG systems, but when the answers look wrong it is hard to tell why they failed. It is not obvious whether the root cause is retrieval collapse, chunk drift, embedding mismatch, or something in the orchestration logic. People often end up debugging by trial and error on raw logs instead of following a structured checklist.
Describe the solution you'd like
I would like to contribute (or help design) a small example that uses Pathway to stream RAG diagnostics using the WFGY 16-problem map:
https://github.com/onestardao/WFGY/blob/main/ProblemMap/README.md
The example could show a simple text RAG pipeline and log, for each query, which WFGY problem codes are suspected (for example No.1 hallucination and chunk drift, No.2 retrieval interpretation collapse, No.5 semantic vs embedding mismatch, and so on). The implementation can stay inside Pathway primitives, with a light layer of heuristics on top of existing logs and metrics.
Describe alternatives you've considered
Right now I run WFGY diagnostics separately in notebooks and external evaluation tools, then manually correlate those results with Pathway logs. This works for my own experiments, but it is not reproducible for other users and it does not show up as an official example in the docs.
Additional context
WFGY and the 16-problem map are MIT licensed and already referenced by several research groups, for example:
- Harvard MIMS Lab ToolUniverse
- University of Innsbruck Data Science Group (Rankify RAG toolkit)
- QCRI LLM Lab Multimodal RAG Survey
If you feel this fits Pathway, I am happy to adapt the taxonomy and open a PR as a self contained example or small docs page, without adding any heavy dependencies.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading ProblemMap/README.md to understand the WFGY taxonomy and compare it with the Pathway logs and metrics described in the issue. Done means a self-contained Pathway example or small documentation page that reports suspected problem codes for each query without heavy dependencies.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- documentation, machine-learning, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100