Langfuse v4 events_only mode silently drops all Dify traces (legacy ingestion API)
- Dominant language
- TypeScript
- Stars
- 156k
- Forks
- 24.6k
- Avg merge
- 22h 9m
- Merged PRs (30d)
- 610
Description
## Self Checks
- This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general).
- I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones.
- I confirm that I am using English to submit this report.
## Dify version
1.17.0 (also reproduces on current `main`)
## Cloud or Self Hosted
Self Hosted (Docker/Helm), Langfuse self-hosted v4 (tested 4.10-4.18)
## Steps to reproduce
1. Self-host Langfuse v4 with the documented end-state configuration: `LANGFUSE_MIGRATION_V4_WRITE_MODE=events_only` (the default) and `LANGFUSE_MIGRATION_V4_NATIVE_OTEL_BEHAVIOUR=direct`.
2. Configure Langfuse tracing for any Dify app and run a chat / workflow.
3. Look for the trace in Langfuse.
## ✔️ Expected Behavior
Traces (message, workflow, tool, moderation, dataset retrieval, suggested question, conversation-name generation) appear in Langfuse.
## ❌ Actual Behavior
No traces ever appear, with no error surfaced in Dify. The provider (`api/providers/trace/trace-langfuse/src/dify_trace_langfuse/langfuse_trace.py`) posts every write as legacy batch events (`IngestionEvent_TraceCreate` / `SpanCreate` / `GenerationCreate`) to `POST /api/public/ingestion`. Per the [Langfuse v3→v4 migration guide](https://langfuse.com/self-hosting/upgrade/upgrade-guides/upgrade-v3-to-v4#api-changes), in `events_only` mode that endpoint keeps returning `207` but answers `400` for every event type except `score-create`/`sdk-log` — so Dify's writes are rejected per-event while the batch call itself "succeeds", and the traces are silently lost. Keeping Langfuse in `dual` write mode works but forces double writes into ClickHouse indefinitely and is documented as a migration state, not an end state.
The fix is to write through the Langfuse Python SDK v4 OpenTelemetry path (`/api/public/otel`), which is the only supported v4 write path. Notes for anyone picking this up: the SDK has no public API for backdated span start times (see langfuse/langfuse#9404), so post-hoc logging of completed executions needs raw OTel spans created via the SDK tracer, and deterministic trace/observation ids (Dify upserts on message/workflow-run/node-execution ids) need a custom OTel id generator.
I have a working implementation, verified end-to-end against Langfuse 4.18 in `events_only` + `direct` across all seven trace flows — PR to follow.
Contributor guide
Research direction
Start with provider/api/providers/trace/trace-langfuse/src/dify_trace_langfuse/langfuse_trace.py and trace its legacy batch writes to /api/public/ingestion. Review the Langfuse Python SDK v4 OpenTelemetry path, including raw spans and deterministic IDs, then verify all seven trace flows against Langfuse 4.18 in events_only plus direct mode; done means traces appear rather than being rejected per event.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, observability
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100