MaartenGr / MaartenGr/BERTopic

Add test coverage around OpenAI(BaseRepresentation)

Open
#2,051 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.8k
Forks
920
Avg merge
22h 24m
Merged PRs (30d)
5

Description

# Goal

I would like to make it easier to verify the behavior of the `OpenAI` class by adding test coverage.

# Proposal

Test coverage will aim at the following scenarios—both for their happy path and edge cases. Although, additional tests may be added in the implementation phase.

Cases to cover:
- Chat
- Chat with backoff
- Completions
- Completions with backoff

Tests will be divided into happy path and edge case scenarios to make nominal and exceptional behaviors clear to the reader.
In order to hold the design of the `OpenAI` class static, I would like to pursue a mocking strategy for the `openai.OpenAI.client`. I would like to create two drafts with one using [OpenAI-Responses package](https://github.com/mharrisb1/openai-responses-python?tab=readme-ov-file) and the other using [unittest.mock.MagicMock](https://docs.python.org/3/library/unittest.mock.html#unittest.mock.MagicMock). I will defer to your preference on readability between the two libraries.

### Considerations

Here are some tradeoffs that I considered when drafting this proposal:

The inherent risk of mock based testing is that the underlying API can diverge from the mock's behavior. However, it allows us to have confidence of the integration on the edges of our system.

When selecting a mocking strategy, I considered using either a built in Python package or a specialized package that provides a simpler interface. The [OpenAI-Responses package](https://github.com/mharrisb1/openai-responses-python?tab=readme-ov-file) can make it easy to mock the OpenAI behavior, but in the scenario of an API functionality drifts, the update loop may be longer or completely blocked by the maintainer of that package. In terms of the built in `MagicMock` class, mocking will be verbose and difficult to reason about, coupling your codebase to the nested attributes of the API client. Comprehension can be improved using construction patterns but that adds to the maintenance cost.

Despite the downsides, having an additional automated quality check outside of manual testing can make both contributions and reviews easier.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the OpenAI class and its openai.OpenAI.client calls, then review existing test conventions. Add tests for chat and completions, including their backoff, happy-path, and edge-case behavior, using a mocking strategy; done means these scenarios are covered without changing the class design.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Refactor
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.