OpenFn / OpenFn/lightning

Add user feedback reporting to AI messages

Open
#4,875 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

AI
Dominant language
Elixir
Stars
296
Forks
86
Avg merge
1d 13h
Merged PRs (30d)
50

Description

Describe the bug
We used to have a user feedback collection button for AI assistant messages, set up here: https://github.com/OpenFn/lightning/pull/2831 . We didn't re-implement it with the React collab-editor AI chat.

Expected behavior
The feedback collection opened a modal with a survey/feedback form, wired to PostHog. However, now that we have Langfuse set up, ideally we'd use that to collect the feedback directly on there. This may need a trace ID, which is currently only handled by Apollo. It might be possible with just the sessionId, which Lightning already handles.

The feedback collection would ideally have thumbs up and thumbs down buttons, and an optional text field. It should identify the specific message the user clicked on, and not the entire session.

We'd need to think through what to do if the user has opted out of metrics, and so there will be no existing session to attach the message(s) to. Additionally, Langfuse is now only tracing core contributor messages, as it isn't self-hosted yet.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing PR 2831 and the current React collab-editor AI chat, then trace how Apollo exposes message or session identifiers and how Langfuse receives traces. Clarify the behavior for metrics opt-outs and untraced messages before implementation. Done means message-specific thumbs up/down feedback with optional text is collected through the chosen service.

Written by the indexing model from the issue text.

Assessment

Tech stack
elixir, graphql, react
Domain
analytics, frontend, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.