microsoft / microsoft/foundry-dev-tools

Creating custom code evaluator drops leading and trailing quotation marks in parameter column

Open
#194 1 comment 0 reactions 1 assignee View on GitHub

@qinezh is already working on this.

Since Apr 29, 2025.

bug upstream
Dominant language
JavaScript
Stars
2.1k
Forks
260
Avg merge
42m
Merged PRs (30d)
29

Description

If the value of a column starts AND ends with the quotation character (") the quotation characters are dropped when sent to the evaluator's custom python function. I wanted to create an evaluator that compared the ground_truth to the response columns to see if the only difference was starting and ending quotation marks. Unfortunately the quotation marks are stripped away when sent to the custom evaluator so the difference was not able to be detected.

If the value only starts or ends with a quotation mark, it is not dropped. It is only when it starts and ends with the quotation mark that the problem occurs.

Interestingly, the built in code evaluators (BLEU, F1 Score, GLEU and METEOR) all seem to be able to see that the result is not the same as the ground_truth so they must be receiving the leading and trailing quotation marks.

Expected behavior:
The values sent to the parameters in the python function for a custom code evaluator should be exactly what they are in the dataset's cell with no characters stripped away.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.