langfuse / langfuse/langfuse-python
batch_evaluation fails on self-hosted v4 events_only deployments (uses unavailable v3 read endpoints)
@hassiebp is already working on this.
Since Sep 5, 2026.
- Dominant language
- Python
- Stars
- 467
- Forks
- 348
- Avg merge
- 12h 16m
- Merged PRs (30d)
- 27
Description
Environment
- Langfuse server: self-hosted v4 (
docker.io/langfuse/langfuse:4), default write modeevents_only - Python SDK: 4.14.4 (also verified against 4.15.1 source)
What happened
Running batch evaluation against existing traces fails. The runner fetches items via endpoints that are unavailable on v4 events_only deployments:
scope="traces"→client.api.trace.list→GET /api/public/tracesscope="observations"→ legacyobservations_v1→GET /api/public/observations
Both return:
"This endpoint is not available on deployments running in Langfuse v4 events_only mode."
Root cause
BatchEvaluationRunner._fetch_batch_with_retry (langfuse/batch_evaluation.py, line 1139 in v4.14.4) still calls the v3 read APIs. Per the v3→v4 migration guide, reads should go through GET /api/public/v2/observations (client-side grouping by traceId, root observation for trace-level io).
Expected
batch_evaluation works on events_only deployments, e.g. by fetching via the v2 observations API.
Workaround
We currently run our own thin loop: fetch v2 observations grouped by traceId → evaluate → create_score (score-create ingestion works fine on v4).
Happy to contribute a PR if the v2-based approach sounds right — glad to align on the design first.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.