googleforgames / googleforgames/GenAI-quickstart

Os Environment Variables and LLM Calls Should be Mocked for Tests

Open
#28 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
51
Forks
21
PR merge metrics
No merged PRs in 30d

Description

**What happened?**

When running `pytest -v` in a local virtualenv on the `test_main.py` files:

- genai_api/src/test_main.py fails when running `pytest` because the OS environment variables are not in place.
- api/vertex_*/src/test_main.py fail when running `pytest` because there is no mock for `Google_Cloud_GenAI`, so the test tries to connect to a real project and call a LLM in that project.
- api/vertex_*/src/test_main.py do not have the correct `expected_response` since they return a single text string `response.text` and not a json object `{'mocked_key': 'mocked_value'}`.

**What you expected to happen:**

Running the unit tests should not require setting up environment variables or calls to an external service.

**How to reproduce it (as minimally and precisely as possible):**

For the genai_api test:

1. Create and activate a python virtual environment
2. Run `pytest` in `(myvirtualenv) me@me:~/GenAI-quickstart/genai/api/genai_api/src$ pytest -v`

For example on the vertex_chat_api test:
1. Create and activate a python virtual environment
2. Run `pytest` in `(myvirtualenv) me@me:~/GenAI-quickstart/genai/api/vertex_chat_api/src$ pytest -v` (you'll notice this fails with warning that the `403 Vertex AI API has not been used in project`)
3. Change `project_id_response.text` to the name of your Google Cloud Project. https://github.com/googleforgames/GenAI-quickstart/blob/205ba78aa65782f00a8654e31c88efdc4fdaded6/genai/api/vertex_chat_api/src/main.py#L45
4. Run `pytest` again in `(myvirtualenv) me@me:~/GenAI-quickstart/genai/api/vertex_chat_api/src$ pytest -v`. Now the test fails with `AssertionError: assert 'test response' == {'mocked_key': 'mocked_value'}`.

**Anything else we need to know?:**

The value `test response` is coming from the actual call to the LLM. If you run the GenAI Quickstart cluster and navigate to the http://${EXT_IP}/genai_docs (following instructions on the [main readme](https://github.com/googleforgames/GenAI-quickstart/tree/main)) and enter the same payload prompt as the test `"prompt": "test prompt"`, the response is `test response`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.