langchain-ai / langchain-ai/langsmith-java
Link traced evaluation runs to their dataset examples
Nobody has claimed this yet.
- Dominant language
- Kotlin
- Stars
- 33
- Forks
- 8
- Avg merge
- 8d 3h
- Merged PRs (30d)
- 3
Description
Problem
Successful traced evaluation runs set referenceExampleId only when EvaluateErrorHandling.LOG is configured. When using IGNORE, successful runs are not linked to the dataset example that produced them.
This weakens traceability and makes it harder to inspect evaluation results by example in LangSmith.
Expected behavior
All successful traced evaluation runs should include the ID of their source dataset example, regardless of error-handling mode.
Proposed fix
Always set referenceExampleId to example.id() when creating the evaluation target’s TraceConfig.
Acceptance criteria
- Traced runs include
reference_example_idforLOGandIGNOREmodes. - Existing error-handling behavior is unchanged.
- Add regression coverage for a successful evaluation using
EvaluateErrorHandling.IGNORE.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the code that creates the evaluation target’s TraceConfig and compare how referenceExampleId is set for EvaluateErrorHandling.LOG versus IGNORE. Use example.id() as the expected source identifier, then add regression coverage for a successful IGNORE evaluation. Done means traced runs include reference_example_id in both modes while existing error-handling behavior remains unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kotlin
- Domain
- api
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100