Automated feedback loop

Think about an automatic mechanism in which we can use the feedback we get from the users back in our evaluation dataset.

Use it as part of our dataset ground truth (good or bad )

Tasks

• DeepEval using explicit feedback https://docs.confident-ai.com/docs/confident-ai-human-feedback

For this, we need to institute more metrics (tracked in #18 )

Explicit feedback

• Thumbs up/ thumbs down. But this is rare.

Implicit feedback

• Inferred from user behavior, such as response time, follow-up questions, or even actions like copying and pasting responses. Implicit feedback is far more abundant and can provide valuable insights into how well the LLM is performing

1. https://towardsdatascience.com/how-to-make-the-most-out-of-llm-production-data-simulated-user-feedback-843c444febc7
2. https://www.nebuly.com/blog/llm-feedback-loop