Azure / Azure/azure-openai-benchmark

Decoupling Throttling retries logic & Throttling backoff logic.

Open
#34 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
154
Forks
76
PR merge metrics
No merged PRs in 30d

Description

As checked from the code, we have 2 retry logics for retrying 429 errors:

image

File: oairequester.py
Class: OAIRequester
Function: _call

& Currently both these two retry logics are coupled together.

This violates Separation of Concerns principle.
Also, not sure if the end user would always want that - i.e. multiple retries for the same request from two different logics.
For example, If Backoff strategy is specified by the user, then the same 429 request which was retried in previous code snippet based on RETRY_AFTER_MS_HEADER will again be retried. multiple times again!

_Solution Suggestion_
It would be a good idea that (just like backoff) to also parametrize the retry logic from command line arguments.
So that the end user could then accordingly adjust the behavior of retries as per their requirement.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in oairequester.py at OAIRequester._call and trace both 429 retry paths, including the RETRY_AFTER_MS_HEADER handling and configured backoff behavior. Determine how command-line arguments currently configure backoff, then define separate retry controls so one request is not retried by both mechanisms. Done means the retry strategies are independently configurable and their interaction is covered by the project’s existing checks.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend-api-design, cli
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.