ianarawjo / ianarawjo/ChainForge
Inconsistent response Content in evaluate Function When Toggling Python Evaluator Sandbox
- Dominant language
- TypeScript
- Stars
- 3k
- Forks
- 256
- Avg merge
- 4h 40m
- Merged PRs (30d)
- 22
Description
**Describe the bug**
When using the Python evaluator, enabling or disabling the sandbox causes the `response` parameter passed into the `evaluate` function to contain different content. This results in inconsistent evaluation behavior depending on the sandbox setting. The structure of `response` does not match what is described in the [[official documentation](https://chainforge.ai/docs/evaluation/)](https://chainforge.ai/docs/evaluation/).
**To Reproduce**
Steps to reproduce the behavior:
1. Create a Python evaluator.
2. Disable the sandbox environment.
3. Print out various attributes of the `response` object in the `evaluate` function.
4. You will observe a set of integers being printed, rather than the documented `.text`, `.raw`, etc.
**Expected behavior**
The `response` parameter should have a consistent structure, regardless of whether the sandbox is enabled or not. It should follow the documented interface.
**Screenshots**
**Environment:**
* OS: Ubuntu 11 (server), Windows 11 (client)
* Browser: Edge 138.0.3351.95
* Python: 3.12
**Additional context**
This inconsistency makes it difficult to write portable evaluator scripts. It would be helpful to unify the `response` object structure between sandboxed and non-sandboxed execution.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.