langgenius / langgenius/dify

Langfuse cannot use the Evaluators Run on live incoming observation mode

Open
#38,805 3 comments 1 reaction 0 assignees View on GitHub
🐞 bug 1.15.0 project#dify
Dominant language
TypeScript
Stars
156k
Forks
24.6k
Avg merge
22h 9m
Merged PRs (30d)
610

Description

### Self Checks

- [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542).
- [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general).
- [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones.
- [x] I confirm that I am using English to submit this report, otherwise it will be closed.
- [x] 【中文用户 & Non English User】请使用英语提交,否则会被关闭 :)
- [x] Please do not modify this template :) and fill in all the required fields.

### Dify version

1.15.0

### Cloud or Self Hosted

Self Hosted (Docker)

### Steps to reproduce

After configuring the latest version of langfuse in the workflow, I created an observer and set the observation mode to "Run based on real-time incoming observation data". However, even after enabling Evaluators, I am still unable to track the node logs of dify.
If logs are manually created using the langfuse Python SDK, they can be inspected by observers. However, if logs are created using dify, the created log records can be traced by Tracing but cannot be inspected by observers.
I ensure that it is definitely not an observer filtering issue. If the observer uses the legacy evaluator, it should also be able to capture the single invocation log of dify.

dify : 1.15.0
langfuse : v3.211.0
windows 11
docker deploy

### ✔️ Expected Behavior

Langfuse can capture node logs using Evaluators for scoring

### ❌ Actual Behavior

No node logs were observed among the evaluators of langfuse

Contributor guide

Open the contributing guide

Research direction

Reproduce the workflow with Dify 1.15.0, Langfuse v3.211.0, and the Docker deployment, comparing Dify-created logs with logs created through the Langfuse Python SDK. Investigate why Tracing can see Dify node logs while real-time Evaluators cannot; done when Evaluators capture those node logs for scoring.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.