langchain-ai / langchain-ai/langsmith-java

Link traced evaluation runs to their dataset examples

Open Beginner friendly
#210 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Kotlin
Stars
33
Forks
8
Avg merge
8d 3h
Merged PRs (30d)
3

Description

Problem

Successful traced evaluation runs set referenceExampleId only when EvaluateErrorHandling.LOG is configured. When using IGNORE, successful runs are not linked to the dataset example that produced them.

This weakens traceability and makes it harder to inspect evaluation results by example in LangSmith.

Expected behavior

All successful traced evaluation runs should include the ID of their source dataset example, regardless of error-handling mode.

Proposed fix

Always set referenceExampleId to example.id() when creating the evaluation target’s TraceConfig.

Acceptance criteria

  • Traced runs include reference_example_id for LOG and IGNORE modes.
  • Existing error-handling behavior is unchanged.
  • Add regression coverage for a successful evaluation using EvaluateErrorHandling.IGNORE.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the code that creates the evaluation target’s TraceConfig and compare how referenceExampleId is set for EvaluateErrorHandling.LOG versus IGNORE. Use example.id() as the expected source identifier, then add regression coverage for a successful IGNORE evaluation. Done means traced runs include reference_example_id in both modes while existing error-handling behavior remains unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
kotlin
Domain
api
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.