microsoft / microsoft/foundry-dev-tools
Creating custom code evaluator drops leading and trailing quotation marks in parameter column
@qinezh is already working on this.
Since Apr 29, 2025.
- Dominant language
- JavaScript
- Stars
- 2.1k
- Forks
- 260
- Avg merge
- 42m
- Merged PRs (30d)
- 29
Description
If the value of a column starts AND ends with the quotation character (") the quotation characters are dropped when sent to the evaluator's custom python function. I wanted to create an evaluator that compared the ground_truth to the response columns to see if the only difference was starting and ending quotation marks. Unfortunately the quotation marks are stripped away when sent to the custom evaluator so the difference was not able to be detected.
If the value only starts or ends with a quotation mark, it is not dropped. It is only when it starts and ends with the quotation mark that the problem occurs.
Interestingly, the built in code evaluators (BLEU, F1 Score, GLEU and METEOR) all seem to be able to see that the result is not the same as the ground_truth so they must be receiving the leading and trailing quotation marks.
Expected behavior:
The values sent to the parameters in the python function for a custom code evaluator should be exactly what they are in the dataset's cell with no characters stripped away.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.