MaartenGr / MaartenGr/BERTopic
Add test coverage around OpenAI(BaseRepresentation)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
# Goal
I would like to make it easier to verify the behavior of the `OpenAI` class by adding test coverage.
# Proposal
Test coverage will aim at the following scenarios—both for their happy path and edge cases. Although, additional tests may be added in the implementation phase.
Cases to cover:
- Chat
- Chat with backoff
- Completions
- Completions with backoff
Tests will be divided into happy path and edge case scenarios to make nominal and exceptional behaviors clear to the reader.
In order to hold the design of the `OpenAI` class static, I would like to pursue a mocking strategy for the `openai.OpenAI.client`. I would like to create two drafts with one using [OpenAI-Responses package](https://github.com/mharrisb1/openai-responses-python?tab=readme-ov-file) and the other using [unittest.mock.MagicMock](https://docs.python.org/3/library/unittest.mock.html#unittest.mock.MagicMock). I will defer to your preference on readability between the two libraries.
### Considerations
Here are some tradeoffs that I considered when drafting this proposal:
The inherent risk of mock based testing is that the underlying API can diverge from the mock's behavior. However, it allows us to have confidence of the integration on the edges of our system.
When selecting a mocking strategy, I considered using either a built in Python package or a specialized package that provides a simpler interface. The [OpenAI-Responses package](https://github.com/mharrisb1/openai-responses-python?tab=readme-ov-file) can make it easy to mock the OpenAI behavior, but in the scenario of an API functionality drifts, the update loop may be longer or completely blocked by the maintainer of that package. In terms of the built in `MagicMock` class, mocking will be verbose and difficult to reason about, coupling your codebase to the nested attributes of the API client. Comprehension can be improved using construction patterns but that adds to the maintenance cost.
Despite the downsides, having an additional automated quality check outside of manual testing can make both contributions and reviews easier.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the OpenAI class and its openai.OpenAI.client calls, then review existing test conventions. Add tests for chat and completions, including their backoff, happy-path, and edge-case behavior, using a mocking strategy; done means these scenarios are covered without changing the class design.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing-qa
- Issue type
- Refactor
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100