langgenius / langgenius/dify

Langfuse v4 events_only mode silently drops all Dify traces (legacy ingestion API)

Open
#41,542 0 comments 2 reactions 0 assignees View on GitHub
1.17.0
Dominant language
TypeScript
Stars
156k
Forks
24.6k
Avg merge
22h 9m
Merged PRs (30d)
610

Description

## Self Checks

- This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general).
- I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones.
- I confirm that I am using English to submit this report.

## Dify version

1.17.0 (also reproduces on current `main`)

## Cloud or Self Hosted

Self Hosted (Docker/Helm), Langfuse self-hosted v4 (tested 4.10-4.18)

## Steps to reproduce

1. Self-host Langfuse v4 with the documented end-state configuration: `LANGFUSE_MIGRATION_V4_WRITE_MODE=events_only` (the default) and `LANGFUSE_MIGRATION_V4_NATIVE_OTEL_BEHAVIOUR=direct`.
2. Configure Langfuse tracing for any Dify app and run a chat / workflow.
3. Look for the trace in Langfuse.

## ✔️ Expected Behavior

Traces (message, workflow, tool, moderation, dataset retrieval, suggested question, conversation-name generation) appear in Langfuse.

## ❌ Actual Behavior

No traces ever appear, with no error surfaced in Dify. The provider (`api/providers/trace/trace-langfuse/src/dify_trace_langfuse/langfuse_trace.py`) posts every write as legacy batch events (`IngestionEvent_TraceCreate` / `SpanCreate` / `GenerationCreate`) to `POST /api/public/ingestion`. Per the [Langfuse v3→v4 migration guide](https://langfuse.com/self-hosting/upgrade/upgrade-guides/upgrade-v3-to-v4#api-changes), in `events_only` mode that endpoint keeps returning `207` but answers `400` for every event type except `score-create`/`sdk-log` — so Dify's writes are rejected per-event while the batch call itself "succeeds", and the traces are silently lost. Keeping Langfuse in `dual` write mode works but forces double writes into ClickHouse indefinitely and is documented as a migration state, not an end state.

The fix is to write through the Langfuse Python SDK v4 OpenTelemetry path (`/api/public/otel`), which is the only supported v4 write path. Notes for anyone picking this up: the SDK has no public API for backdated span start times (see langfuse/langfuse#9404), so post-hoc logging of completed executions needs raw OTel spans created via the SDK tracer, and deterministic trace/observation ids (Dify upserts on message/workflow-run/node-execution ids) need a custom OTel id generator.

I have a working implementation, verified end-to-end against Langfuse 4.18 in `events_only` + `direct` across all seven trace flows — PR to follow.

Contributor guide

Open the contributing guide

Research direction

Start with provider/api/providers/trace/trace-langfuse/src/dify_trace_langfuse/langfuse_trace.py and trace its legacy batch writes to /api/public/ingestion. Review the Langfuse Python SDK v4 OpenTelemetry path, including raw spans and deterministic IDs, then verify all seven trace flows against Langfuse 4.18 in events_only plus direct mode; done means traces appear rather than being rejected per event.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, observability
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.