pathwaycom / pathwaycom/pathway

[Proposal] Example: streaming RAG diagnostics with the WFGY 16-problem map in Pathway

Open
#194 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
62.3k
Forks
1.7k
PR merge metrics
No merged PRs in 30d

Description

Is your feature request related to a problem? Please describe.

Many Pathway users build RAG systems, but when the answers look wrong it is hard to tell why they failed. It is not obvious whether the root cause is retrieval collapse, chunk drift, embedding mismatch, or something in the orchestration logic. People often end up debugging by trial and error on raw logs instead of following a structured checklist.

Describe the solution you'd like

I would like to contribute (or help design) a small example that uses Pathway to stream RAG diagnostics using the WFGY 16-problem map:

https://github.com/onestardao/WFGY/blob/main/ProblemMap/README.md

The example could show a simple text RAG pipeline and log, for each query, which WFGY problem codes are suspected (for example No.1 hallucination and chunk drift, No.2 retrieval interpretation collapse, No.5 semantic vs embedding mismatch, and so on). The implementation can stay inside Pathway primitives, with a light layer of heuristics on top of existing logs and metrics.

Describe alternatives you've considered

Right now I run WFGY diagnostics separately in notebooks and external evaluation tools, then manually correlate those results with Pathway logs. This works for my own experiments, but it is not reproducible for other users and it does not show up as an official example in the docs.

Additional context

WFGY and the 16-problem map are MIT licensed and already referenced by several research groups, for example:

  • Harvard MIMS Lab ToolUniverse
  • University of Innsbruck Data Science Group (Rankify RAG toolkit)
  • QCRI LLM Lab Multimodal RAG Survey

If you feel this fits Pathway, I am happy to adapt the taxonomy and open a PR as a self contained example or small docs page, without adding any heavy dependencies.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading ProblemMap/README.md to understand the WFGY taxonomy and compare it with the Pathway logs and metrics described in the issue. Done means a self-contained Pathway example or small documentation page that reports suspected problem codes for each query without heavy dependencies.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
documentation, machine-learning, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.