Tasks should be "indeterminate" if the API response has any errors
- Dominant language
- Python
- Stars
- 123
- Forks
- 25
- PR merge metrics
- No merged PRs in 30d
Description
### Big idea
If there isn't some mechanism to surface API errors, then there can be type I/II errors that giving false signal to the benchmark metrics.
### Details
See https://github.com/thedatamates/ade-bench/issues/52 for some examples of API success or failure.
For example, a successful Gemini API call when `--agent gemini --model SPECIFIED-MODEL-NAME-HERE` should look like the following:
```json
{
"response": "...",
"stats": {
"models": {
"SPECIFIED-MODEL-NAME-HERE": {
"api": {
"totalErrors": 0,
}
}
}
}
```
Otherwise, the task should be considered "indeterminate" (not enough information to know if the task failed or succeeded).
There should be some way that the user knows this task needs troubleshooting until it can be re-executed without an API failure.
Contributor guide
Assessment
This issue has not been assessed yet.