ianarawjo / ianarawjo/ChainForge

Inconsistent response Content in evaluate Function When Toggling Python Evaluator Sandbox

Open
#371 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
3k
Forks
256
Avg merge
4h 40m
Merged PRs (30d)
22

Description

**Describe the bug**
When using the Python evaluator, enabling or disabling the sandbox causes the `response` parameter passed into the `evaluate` function to contain different content. This results in inconsistent evaluation behavior depending on the sandbox setting. The structure of `response` does not match what is described in the [[official documentation](https://chainforge.ai/docs/evaluation/)](https://chainforge.ai/docs/evaluation/).

**To Reproduce**
Steps to reproduce the behavior:

1. Create a Python evaluator.
2. Disable the sandbox environment.
3. Print out various attributes of the `response` object in the `evaluate` function.
4. You will observe a set of integers being printed, rather than the documented `.text`, `.raw`, etc.

**Expected behavior**
The `response` parameter should have a consistent structure, regardless of whether the sandbox is enabled or not. It should follow the documented interface.

**Screenshots** Image

**Environment:**

* OS: Ubuntu 11 (server), Windows 11 (client)
* Browser: Edge 138.0.3351.95
* Python: 3.12

**Additional context**
This inconsistency makes it difficult to write portable evaluator scripts. It would be helpful to unify the `response` object structure between sandboxed and non-sandboxed execution.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.