dbt-labs / dbt-labs/ade-bench

Tasks should be "indeterminate" if the API response has any errors

Open
#57 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
123
Forks
25
PR merge metrics
No merged PRs in 30d

Description

### Big idea

If there isn't some mechanism to surface API errors, then there can be type I/II errors that giving false signal to the benchmark metrics.

### Details

See https://github.com/thedatamates/ade-bench/issues/52 for some examples of API success or failure.

For example, a successful Gemini API call when `--agent gemini --model SPECIFIED-MODEL-NAME-HERE` should look like the following:
```json
{
"response": "...",
"stats": {
"models": {
"SPECIFIED-MODEL-NAME-HERE": {
"api": {
"totalErrors": 0,
}
}
}
}
```

Otherwise, the task should be considered "indeterminate" (not enough information to know if the task failed or succeeded).

There should be some way that the user knows this task needs troubleshooting until it can be re-executed without an API failure.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.