Classification : Implement Batch API for model evaluation

Open
#345 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
42/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Stale
Tech stack
python
Domain
api, backend-api-design, machine-learning

Research direction

Review the current synchronous Responses API evaluation flow in the repository, then consult the linked design document for the proposed batch workflow. Done means evaluation queries are submitted through the Batch API, predictions are read from the batch output file, and those predictions are compared with the corresponding true labels.

Written by the indexing model from the issue text.

Description

enhancement

Current Behavior
For evaluation of fine-tuned models, we currently use the Responses API synchronously to generate predictions. These predictions are then compared against the true labels of the corresponding questions. However, this synchronous setup is difficult to manage and would require building a complex system to make the whole process asynchronous and more reliable.

Proposed Improvement
Instead of relying on the synchronous approach, we can leverage the OpenAI Batch API. By sending all evaluation queries together as a batch, we can retrieve the predictions from the batch output file and then compare them against the true labels. This would simplify the workflow and make the evaluation process more scalable and efficient.

document to refer : https://docs.google.com/document/d/1y2WLRUm2dZaQMetvb31XpDH5HqteRe4UNpziG8pq608/edit?tab=t.lbfxo8c09bqd#bookmark=id.1ib6z5f31fld

Dominant language
Python
Stars
18
Forks
10
Avg merge
2d 20h
Merged PRs (30d)
14

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from ProjectTech4DevAI/kaapi-backend

All issues in ProjectTech4DevAI/kaapi-backend

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.